I am often bemoaning how HCI became a purely empirical field. Even in settings where statistical theory is clearly relevant to defining what we want humans to achieve (e.g., better decisions from use of model predictions or visualizations), there's a tendency to see theory as irrelevant. Meanwhile, many user studies, when you analyze them theoretically, are more or less dead in the water... what they hoped to find was unlikely from the get-go.
Interesting post! At some points it felt like you were arguing that CS is evolving into a "purely experimental/empirical" science, which I would push back against. I see it more as CS maturing into a place where theoretical/conceptual work is no longer the ultimate goal, but another piece of the puzzle that should take into account and communicate well with the empirical evidence and work. Of course, this might just be my survival instincts as a theoretician kicking in :).
This post certainly reminded me of Recht's post on "Computational Mythmaking", where CS being a discipline closer to mathematics was seen as a signal of value of the field. It does feel we are navigating towards a more balanced view of CS nowadays.
Psychology has spent a long time thinking about how to do experimental empiricism on abstract concepts that aren't actually observable. Some of this might be useful for Comp Sci; maybe y'all should learn to talk about constructs and operationalization and validity and all that.
But on the other hand, Machine Learning has a major advantage over psychology, and also every other social science: you can tell when things work! When an ML preprint makes some wacky claims based haphazard experiments, practitioners can just go on their own computers and see what's up.
In social science, by contrast, the only timely and unambiguous benchmark for success is "do other scientists talk about it". All that philosophy of empiricism stuff provides a scientific-ish structure to the conversation, but it doesn't prevent people from talking nonsense.
I am often bemoaning how HCI became a purely empirical field. Even in settings where statistical theory is clearly relevant to defining what we want humans to achieve (e.g., better decisions from use of model predictions or visualizations), there's a tendency to see theory as irrelevant. Meanwhile, many user studies, when you analyze them theoretically, are more or less dead in the water... what they hoped to find was unlikely from the get-go.
Not to mention that the huge fraction of user studies in HCI are underpowered...
Interesting post! At some points it felt like you were arguing that CS is evolving into a "purely experimental/empirical" science, which I would push back against. I see it more as CS maturing into a place where theoretical/conceptual work is no longer the ultimate goal, but another piece of the puzzle that should take into account and communicate well with the empirical evidence and work. Of course, this might just be my survival instincts as a theoretician kicking in :).
This post certainly reminded me of Recht's post on "Computational Mythmaking", where CS being a discipline closer to mathematics was seen as a signal of value of the field. It does feel we are navigating towards a more balanced view of CS nowadays.
Psychology has spent a long time thinking about how to do experimental empiricism on abstract concepts that aren't actually observable. Some of this might be useful for Comp Sci; maybe y'all should learn to talk about constructs and operationalization and validity and all that.
But on the other hand, Machine Learning has a major advantage over psychology, and also every other social science: you can tell when things work! When an ML preprint makes some wacky claims based haphazard experiments, practitioners can just go on their own computers and see what's up.
In social science, by contrast, the only timely and unambiguous benchmark for success is "do other scientists talk about it". All that philosophy of empiricism stuff provides a scientific-ish structure to the conversation, but it doesn't prevent people from talking nonsense.