Isn't an important issue the rising complexity of "solutions". Feedback loops and individual responsibility requires that somebody actually can have oversight and an understanding of all relevant aspects. If systems get more and more complex, the level of complexity will go beyond human capabilities, which makes it harder to assign individual responsibility (and hence the rise of bureaucracy to deal with it.
However, there is another aspect. Seemingly we humans need to see a human person as responsible, and cannot handle yet that this may not be the case. For example, driver less cars are safer and better "drivers" than many human drivers, yet we are much concerned with their "safety" than with that of humans. One of the reasons for that, at least in my view, is that it is not clear who to assign blame to when a driver-less car does indeed cause an accident.
Oh absolutely. I think systems are increasingly complex, but maybe that’s a problem. Paradoxically maybe we have too many resources and too much knowledge - so can’t see the wood for the trees.
With regard to responsibility - as a lawyer, of course I agree entirely with the notion that humans like to be able to attribute blame to someone (or something). One of my obsessions is with those instances when people attribute blame to objects or animals rather than other humans. It’s really fascinating how we look for other outlets if there isn’t a person to blame. Maybe I need to do a post on the strange law of deodands, something which was only abolished in the 19th C. Yes, it was possible at common law to effectively blame “things” for causing injury! One could laugh - but then one could recall how one swore at the printer and hit it and told it off when it malfunctioned… 😳
The most significant part of this is figuring out what you want and how you're going to get it. If there's no goal or plan in mind, then it's hard for anyone else to help you achieve it, IMHO.
Yes, exactly. You need to know what you want. And sometimes - if you’ve got multiple people driving a project - that’s part of the problem, they’ve all got different aims.
I do indeed know of his ‘Penal Colony’ - read it when I was about 17. It’s freaked me out ever since! The bridge example is a classic example of bad feedback with disastrous consequences…
And another thought about Robodebt (though I am not fully familiar with the detail, so this is more a general thought bubble). One aspect that went wrong was what Type I & Type II error trade off was socially acceptable and which is not.
In criminal law the "better that 10 guilty go free, then an innocent goes to jail" maxim implies that trials are designed to minimize Type I errors (an innocent gets wrongly sent to jail), but takes into account a higher number of Type II errors (guilty go free). In Robodebt, the underlying principle of the algorithm and enforcement seems to have shifted from a focus on minimizing Type I errors (somebody is forced to pay debts they don't own) / accepting Type II errors (many debts were not collected) to a different balance where many more debts were actually collected, while accepting that more erroneous debts were imposed on people.
The issue with Robodebt isn't so much that they changed the trade-off (which did indeed collect some more actual debts), but that the "losers" were what we consider "vulnerable". If the losers would have been rich (or socially stigmatized) then this would not have been perceived as such a scandal.
So for me - it really does matter that they didn’t owe the debt. The system was careless as to any kind of error, and that’s unacceptable. I wouldn’t care if those who lost were rich or socially stigmatised - the same problem would apply - the rule of law requires that we treat all debtors equally, whether vulnerable or not. Moreover to use that form of legal action in illegitimate ways undermines the very action itself, regardless of the victims. Of course, I’m a lawyer, so my perspective might be odd. I’m sure the societal perception might be different…
A few years back, the ATO sent me a notice along the lines of: "We think you omitted these things from your tax filing, we will proceed as if they are correct". Now, my tax account had really stuffed up, but (at least in my recall) the ATO effectively shifted the burden of proof from them to me ("we don't have to prove that you failed to include this, you have to prove that they are not yours").
I don't know all the details (and I am sure there was more to Robodebt than just this), but part of the process seemingly worked the same way, with the department saying "Based on getting ATO data we think you owe us this much, now prove that you do not owe us". Such a shift will obviously result in the type I - type II error rebalancing I mentioned above, but is in itself (at least in my lay opinion) seemingly in line with general legal principles.
It also treats everyone the same, as everyone would have to prove that the debt is not theirs. What I meant by "rich" vs "vulnerable" was how we as society accept who has the burden of proof in such situations, so is a political questions shaping the choice of different legal arrangements.
No, it wasn’t in line with legal principles. So - debt is a special kind of contractual obligation. It’s a certain sum, triggered by the occurrence of a certain event (a date, the completion of something, the failure to do something etc). You can’t just say we’re going to make up what the certain sum is on the basis of something flawed which doesn’t reflect the triggering event. That’s illegal. Tax liability is different - I’m presuming that there’s a specific statutory system for that, although it’s a long time since I’ve done tax law.
Isn't an important issue the rising complexity of "solutions". Feedback loops and individual responsibility requires that somebody actually can have oversight and an understanding of all relevant aspects. If systems get more and more complex, the level of complexity will go beyond human capabilities, which makes it harder to assign individual responsibility (and hence the rise of bureaucracy to deal with it.
However, there is another aspect. Seemingly we humans need to see a human person as responsible, and cannot handle yet that this may not be the case. For example, driver less cars are safer and better "drivers" than many human drivers, yet we are much concerned with their "safety" than with that of humans. One of the reasons for that, at least in my view, is that it is not clear who to assign blame to when a driver-less car does indeed cause an accident.
Oh absolutely. I think systems are increasingly complex, but maybe that’s a problem. Paradoxically maybe we have too many resources and too much knowledge - so can’t see the wood for the trees.
With regard to responsibility - as a lawyer, of course I agree entirely with the notion that humans like to be able to attribute blame to someone (or something). One of my obsessions is with those instances when people attribute blame to objects or animals rather than other humans. It’s really fascinating how we look for other outlets if there isn’t a person to blame. Maybe I need to do a post on the strange law of deodands, something which was only abolished in the 19th C. Yes, it was possible at common law to effectively blame “things” for causing injury! One could laugh - but then one could recall how one swore at the printer and hit it and told it off when it malfunctioned… 😳
The most significant part of this is figuring out what you want and how you're going to get it. If there's no goal or plan in mind, then it's hard for anyone else to help you achieve it, IMHO.
Yes, exactly. You need to know what you want. And sometimes - if you’ve got multiple people driving a project - that’s part of the problem, they’ve all got different aims.
"It’s like a Kafkaesque nightmare"
Indeed. You probably know of his "Penal Colony":
https://en.wikipedia.org/wiki/In_the_Penal_Colony
But "feedback loops" is -- or was -- my bread and butter, my claim to fame and fortune, such as they were and are. And the failures therein:
https://en.wikipedia.org/wiki/Cybernetics#Etymology
But a classic case of that is the "Tacoma Narrows Bridge Collapse; Gallopin' Gertie":
https://www.youtube.com/watch?v=j-zczJXSxnw
Wind and feedback caused the bridge to sway further and further outside its rest position until it eventually collapsed.
I do indeed know of his ‘Penal Colony’ - read it when I was about 17. It’s freaked me out ever since! The bridge example is a classic example of bad feedback with disastrous consequences…
Not many people know of feedback loops, and probably even fewer of classic examples of how they can go wrong. 👍🙂
You might have some interest in this overview article from Matthew Cobb some 10 years ago:
https://www.sciencedirect.com/science/article/pii/S0092867413004534
But ubiquitous in biology and society, a useful concept and technology in many ways.
And another thought about Robodebt (though I am not fully familiar with the detail, so this is more a general thought bubble). One aspect that went wrong was what Type I & Type II error trade off was socially acceptable and which is not.
In criminal law the "better that 10 guilty go free, then an innocent goes to jail" maxim implies that trials are designed to minimize Type I errors (an innocent gets wrongly sent to jail), but takes into account a higher number of Type II errors (guilty go free). In Robodebt, the underlying principle of the algorithm and enforcement seems to have shifted from a focus on minimizing Type I errors (somebody is forced to pay debts they don't own) / accepting Type II errors (many debts were not collected) to a different balance where many more debts were actually collected, while accepting that more erroneous debts were imposed on people.
The issue with Robodebt isn't so much that they changed the trade-off (which did indeed collect some more actual debts), but that the "losers" were what we consider "vulnerable". If the losers would have been rich (or socially stigmatized) then this would not have been perceived as such a scandal.
So for me - it really does matter that they didn’t owe the debt. The system was careless as to any kind of error, and that’s unacceptable. I wouldn’t care if those who lost were rich or socially stigmatised - the same problem would apply - the rule of law requires that we treat all debtors equally, whether vulnerable or not. Moreover to use that form of legal action in illegitimate ways undermines the very action itself, regardless of the victims. Of course, I’m a lawyer, so my perspective might be odd. I’m sure the societal perception might be different…
A few years back, the ATO sent me a notice along the lines of: "We think you omitted these things from your tax filing, we will proceed as if they are correct". Now, my tax account had really stuffed up, but (at least in my recall) the ATO effectively shifted the burden of proof from them to me ("we don't have to prove that you failed to include this, you have to prove that they are not yours").
I don't know all the details (and I am sure there was more to Robodebt than just this), but part of the process seemingly worked the same way, with the department saying "Based on getting ATO data we think you owe us this much, now prove that you do not owe us". Such a shift will obviously result in the type I - type II error rebalancing I mentioned above, but is in itself (at least in my lay opinion) seemingly in line with general legal principles.
It also treats everyone the same, as everyone would have to prove that the debt is not theirs. What I meant by "rich" vs "vulnerable" was how we as society accept who has the burden of proof in such situations, so is a political questions shaping the choice of different legal arrangements.
No, it wasn’t in line with legal principles. So - debt is a special kind of contractual obligation. It’s a certain sum, triggered by the occurrence of a certain event (a date, the completion of something, the failure to do something etc). You can’t just say we’re going to make up what the certain sum is on the basis of something flawed which doesn’t reflect the triggering event. That’s illegal. Tax liability is different - I’m presuming that there’s a specific statutory system for that, although it’s a long time since I’ve done tax law.