What is a data plant?

A Data Plant is where the long-term processes involving data can take place:

• Before data is even usable for “AI”, it often needs to be augmented with human tags or labels.
• Next, to avoid the “garbage-in-garbage-out” phenomenon, even before any AI work is done, data needs to be thoroughly tested to see that it’s making sense (as opposed to: partial, mislabeled, noisy or just not making sense).
• AI per se means training, testing and inference. Each of these business processes happens thousands of times, on different subsets of the same basic data - we’ll consider a representative example in the next section.
• After an AI algorithm munches data, it may very well be wrong. Often users will need to curate AI outputs; such curation is fed back either to the AI algorithm directly or to the developers of the AI algorithm to make a better one.

What business functions does a data plant support?

End-user-bound:

• Data entry (e.g.: media upload, curation/annotation, feedback on AI outputs)
• Piecemeal inference or retrieval of inference results

AI Team-bound:

• Data storage, in bulk for AI-bound data and piecemeal for curation
• Bulk data retrieval for AI testing and training.
• Data analytics for data QA and cleanup.
• Data cataloging and discoverability

Isn’t what you describe a “data platform” or a “tool”?

No, in the same way as a woodworking desk is just one requirement for a workshop and an empty barn is only a part of what it takes to open a warehouse. A hefty amount of iterative business-specific customization is required to make the “can do” into “does do”.

Tools like Spark, Databricks, Snowflake, etc. make a lot of complex operations easier and cheaper, but they still require a continuous and determined architectural effort to make sense.

Garden variety is not enough!

The naive alternative to designing a data plant proactively is letting one grow organically, which means, by the hands of the AI scientists using the data.

However, this approach, which I call the “garden variety data plant” is actually highly hazardous for an organization that wants to succeed. Let me explain why that is the case:

The training of most AI scientists typically lacks two major elements of engineering practice that allow projects to scale up. These elements are:

• Design for operations: Multiple uninspiring but crucial adapters needed to make the plant’s work available for other people. A great example of that is making a data entry UI, which is crucial to make tagging and feedback convenient, but is simply out of scope for the vast majority of AI scientists.
• Design for maintenance: the data plant needs to be maintained part-time by other people, to free the AI scientist to do new things. A huge cultural bias here is the fact that most academic AI work discards all code written for a project once it ends - so a lot of scientists have little experience with making a system maintainable in their absence.

What do you need to do?

Data-centric organizations, in particular AI organizations, need to treat their data plant as a core enabler of their growth.

As an organizational leader, one has to designate a chain of accountability and outcomes appropriate to the organization’s ambitions going forward at least a year.

As an engineering leader, one needs to ask the hard questions about business: What elements of data architecture are aligned with their organization’s realistic needs? When is a product a hyped fad? Is a product appropriate for your organizational scale? What are the implicit assumptions in every path towards fulfilling the data plant product goals? What resources are available for the project? What are the red lines that indicate the project is in trouble? What is to be done in that case?

In the last couple of months I’d been attentively listening to Israel Broadcasting Authority’s podcast called One Song. The subject is always the people behind an Israeli song that ended up being popular - the authors, the performers, the producers, the inspirations - indeed, anyone affected by the song except the general public. Where possible, an interview is conducted or excerpted; if there are no longer enough interviewees and interviews to work with, prominent experts are brought in to fill in.

After tuning into more than 20 episodes, I think I’ve arrived at some observations which are worth sharing.

Inspiration and inception

One very frequent trope in the podcast is that songs are born out of a gag or an in-joke; such is the case with Alma Zohar’s Indian Love Song (a.k.a. Miguel), Kaveret’s Aging Child, T-Slam’s Gimme Rock’N’Roll and more.

More generally, it appears that the creative birth of a majority of great songs required a touch of serendipity; people going out a tiny bit out of their regular trajectories, in a way that happens to most of us on a daily basis, but in some cases - leaving behind a seed that could germinate. However, in a huge number of cases, the seed did not germinate right away! One abundant pattern is authors keeping little boxes with such seeds that they carry with them for decades. In Leah Goldberg’s words:

And then you laughed so hard when I intoned,
That I’ll approach the heavens’ rosy sweetness,
And pick it up, and fold it lest it slips us,
And store it midst the pages of my tome

(Don’t miss out the episode on And There Was Between Us Just The Shining!)

Perspiration and congregation

While boxing and keeping the seed is a skill in itself, it appears that most of the songs discussed require two more factors to thrive.

One aspect of that is simply professionalism - making that seed grow requires a hefty amount of onward iterative experimentation. At some point in thre song’s trajectory, there’s a phase of massiive reworking of details big and small:

• For music: harmony, cadence, beat, groove
• For performance: finding the right singer (which may require the incumbent one to “give up”!)
• For lyrics: coming up with a C-part or incorporating somebody else’s material
• For production - taking sound elements from other contemporary work, sampling, etc.).

However as we all know, such iteration can be frustrating and is likely to never get anywhere. Therefore, an additional prerequisite is needed for success: the incorporation of feedback – specifically, qualified criticism coming from somebody who cares, but is not as emotionally invested as the active creator. A great example of that is in the episode about how Izhar Ashdot countered Rona Keinan’s creative effort on an original English text she wrote as The Earthquake, in particular by inducing her to sing a Hebrew version as The Deluge; after hearing many other stories, it is apparent that while the song’s creators bear a huge deal of the effort, no matter how good the creators are, somebody else – who could be a professional producer or a peer who is not intimately involved with the work – is needed to unleash the creation’s full potential.

Search for an artistic self

The observation about criticism returns me to what is probably the most pervasive aspect of creators featured in the podcast - the search for an artistic self. If previously I imagined the process of coming up with great works of art as a creatot’s singlehanded search for diamonds, I now have a different metaphor in mind – that of a market of people daily transacting with each others using pieces of junk found on the seashore, which for most uninitiated is plain and simple rubbish. However, once in a while, two or three of the people involved manage to cobble something outstanding out of the flotsam, which is what makes them into creators. The new work in turns forms a new artistic self, and the process repeats. This strange loop is as much wonder as anything else in our world.

A superficial look on the recent government-led reforms in Israel may leave the impression of some principled disagreement over the theory of government. However, in essense, the move is an opportunistic grab for power; the mechanism it aims to install is little more than the individual survival of a certain milieu of political operators and their patrons, some of whom are not even residents of Israel.

I’d like to use the opportunity to present some learnings from the Soviet society, which entered a period of unbridled governmental opportunism in 1965-1985; right at its end, the period was labeled as “stagnation” (“zastoy”, застой). It is of that that the great Russian poet Iosif (Joseph) Brodsky wrote in the early 1970s

A person of second-rate times, I proudly
Declare that thoughts I conceived most profoundly,
Were second-rate; I hereby leave them for all generations
As a guide to contending with suffocation

(Partial translation: Uri Yanover)

Economics: waste at multiple levels

Opportunism’s earliest symptom is the government losing motivation to serve anything but niche interests. Inherently what follows is the waste of public goods (such as the budget), followed by a waste of productivity, a waste of good will and cohesion and over time - a waste of lives.

During the Soviet stagnation, the economy simply froze solid in its tracks, exhausted by massive defense spending, high investment/low yield civilian projects and the progressively growing burden of corruption and black market dynamics.

A great example of the waste of lives is the Soviet involvement in Afghanistan, which quickly escalated a proxy war between the Soviet Union and the United States, continuing which was not in the Soviet interest - yet it was allowed to drag on for a decade.

The forecast for Israeli tech should be apparent here: its current assets will be used in a very short sighted way to cover governmental spending, whether directly by excessive taxation or indirectly through inflation and lack of public services. Within a startup generation (5 years), this will drive key people and resources to seek a more equitable locality.

Nepotism/Corruption as a scourge of human capital

Since an opportunist doesn’t care about outcomes, the most advantageous choices for staffing positions are those that involve kickbacks - the consideration could be financial, quid pro quo or familial (nepotism).

In case the position in question does not accommodate a choice that favors the opportunist, it should at least do as little harm as possible. Hence, the opportunist will not dare to bring in new and promising upstarts, lest the opportunist be eclipsed! Even compared with humanity at large, which is pretty cowardly when it comes to fresh faces, opportunist regimes are plain and simple scared of new people coming on, no matter how brilliant.

Remarkably hundreds of thousands of people from the Soviet Union who led lives not matching their respective talents, went on to immigrate and win new places for themselves in new countries. There they had a chance to demonstrate their brilliance, breaking through the bonds imposed by immigration and smashing their own individual glass ceilings; such was the story of my own family. Yet, millions of other brilliant people likely could not or did not immigrate, and remained in a society that wasted their talent and indeed their very lives.

Capable people notice that an opportunistic regime shunts their growth all too quickly; in Brodsky’s words, they feel suffocated. Given even an iota of an opportunity, they will run, the more talent - the quicker.

The practice of patronage and quid pro quo protectionism has always been present in Israeli public life. After a few good decades, with the weakening of public order, that practice is going to come back with vengeance. The heart of Israeli tech companies is small businesses. They are highly susceptible to regulatory changes, public relation disasters, even minor brushes with the law. Even mild and well-considered changes can hurt them; raging rug-pulls such as the one going on now, are murderous.

On a personal level, Israel is already a major exporter of talent for academia and art; doubtlessly this will also be the case for the talent-rich local tech industry.

Normalizing the stench

A necessary aspect of opportunist governance is an adoption of a specific messaging to delay the government’s inevitable fall from popularity and then tries to hold off discontent for as long as possible.

One core strategem of an opportunist government is to progressively take over ways to signal merit.

Consider the following dialogue, which took place between Brodsky - the poet who wrote the epigram to this blog post, and a trial judge, who was soon going to sentence him to forced labor due to “not being a productive member of society”:

J: And what is your occupation in general?
B: Poet, poet-translator.
J: And who recognized you to be a poet? Who put you in the ranks of poets?
B: No one. (Unprovoked) And who put me in the ranks of humanity?
J: Did you study it?
B: What?
J: How to be a poet? Did you attempt to finish an institute of higher learning … where they prepare … teach …
B: I did not think that it is given to one by education.
J: By what then?
B: I think it is … from God.

It is apparent that for the judge it was genuinely inconceivable that Brodsky could be a poet at all without being a member of some association - and of course in the Soviet system, that could only be a state-affiliated association, in which he would be expected to abide by the wishes of the incumbent generation of poets and administrators - a notion that was fundamentally incompatible with Brodsky’s temperament as an artist.

Opportunistic regimes select for people who are numb towards rot. Tech, on the other hand, requires the opposite: people who are aggressively fighting rot to make their business a success. While it is possible for small pockets of excellence to remain in the midst of overwhelming depravity, they will find an increasingly hard time growing; without being able to grow they will cease being a viable target for investment and eventually cease adding up to an industry.

Long-term ideological setup for oppression

While a lot of Westerners have the mental image of the Soviet Union as a society of Communist zombies, this is simply not the case. As anywhere else on the globe, Soviet political operators tended to follow their own personal interests.

Similar to corporate “company value” sets, state ideologies do not aim to provide guidance on which decisions to make, but rather serve to legitimize the decisions made by those in charge.

In the Soviet Union, communism was used as a pretext to justify the state’s various obsessions and to smother discontent regarding the populace’s awful living conditions. The notion of ill-willing foreign bourgeois elites was a convenient pretext for both the brutality of the regime and its hypertrophied imperialistic tendencies. But in truth it was all a concealment mechanism for the regime’s inability to steer itself to safety.

Since the old elites form by far the greatest political threat to the government, the government will need a pseudo-moral mechanism to delegitimize and oppress them Hence, an opportunistic Israeli government will concoct an ideology for preserving itself in power with the elites framed as an Adversary. Quite a lot of the Israeli tech will get hurt because it has a massive representation of the old elites (arriving from academia, industry and finance). As with any type of opression, the onslaught will be cynical, wanton and cruel.

Summary

It’s funny how reading some sources, for example the Russian Wikipedia, one is tempted to assume that the Stagnation was “not as bad as it sounds”, perhaps a little step backward for an otherwise functional society. However what transpired right after it ended lends to a much darker view.

In 1985, a new Soviet leader was elected - Mikhail Gorbachev. He declared that the preceding epoch of stagnation has ended, and an epoch of rebuilding was forthcoming. He had power and he used that to carry out multiple reforms - which were too little and too late. Gorbachev was ousted in 1991, and the Soviet Union was disbanded. So shabby grew the fabric of everyday life in the Soviet Union that most people met the country’s demise with relief - at least it was relatively bloodless.

Opportunism is not new a thing in Israel, and was not a new thing in the Soviet Union. However there’s a point in which a damaged structure will crumble beyond repair; and I hold that with all of the earthquakes, water leaks and intermittent fires, real and imaginary, we are in the vicinity of that point. Whether we’ve passed it already, only time will tell; in the meanwhile I wish all involved best of luck.

I’m planning a long and sad blog post called “The Four Wicked Problems of the Israeli Polis”. However I realized that quite a few of my readers likely are not familiar with what people call a wicked problem, so I decided first to present a couple of interesting yet tractable cases.

What makes a problem wicked? Instead of repeating Wikipedia, let me refer you to the brilliant performance of “Gee, Officer Krupke”, here at the BBC Proms, and originally from Bernstein and Sondheim’s West Side Story musical.

You can find the lyrics here; they are a dialogue of several young gang members with specialists (law enforcement, judicial, therapeutic, social-work) tasked with solving society’s conflict with those young men.

What happens is:

• Each different take on the situation brings about a different narrative. The specialists contradict each other and fail to converge to a single actionable policy.
• The endgame is unclear. Should the young men be “soda jerkers”, because that’s the best society has in store for them? How can their stigma be removed while not condoning their misdeeds?
• The final weight of the system’s failure is borne by the patrolling Officer Krupke who has very few tools to handle it beyond being a forceful brute, and perpetuating a pretty rotten state of affairs on the ground.

All of the above are the hallmarks of a wicked problem. I’d like to present a case why the question of Twitter is also one.

• The underlying engine of Twitter’s income is digital advertising, however a lot of players think that the value of this market is going to decline soon - due to a number of reasons:
  • intrinsic decline of the advertisement market leaders - Google and Facebook - due to advertisement being long-term toxic to their platforms.
  • growing awareness of advertisement fraud and various reporting issues that undercut digital ad company valuation.
• There’s a well established pattern of politicians extorting social media companies to siphon out resources, most typically - the ability to spread vile political propaganda with impunity. This both places a direct tax on all social media companies, and indirectly devalues the entire online industry, perhaps to the point of becoming an existential threat.
• Twitter is not expected to grow to new markets.
• Twitter’s operations incrementally grew to be extremely expensive - requiring hundreds of thousands of servers and thousands of engineers - just to remain a losing player in the advertising game. Had a pivot to a different model been attempted at Twitter’s pre-acquisition scale, Twitter would have run out of cash much earlier than the move could have succeeded.

The above represents a tangle of different forces and pressures: economical, political, social, entrepreneurial and operational. At the point of Twitter’s acquisition in 2022, the sum of forces was pulling the company down, and fast. Accordingly, I tend to interpret Musk’s strategy as an attempt to cut the Gordian knot and repurpose the freed pieces of rope.

In less allegorical terms, Musk is trying to salvage the one thing still going for Twitter - its US user base of its fundamental app experience - and try to reinvent it as a different, inevitably much smaller, company, that might be able to survive and in our new austere age.

It’ll take a few years to see if Musk’s winning for himself. However, the unfortunate reality for a lot of Twitter employees is that they have lost already, and they need to fight for a new source of daily bread as disparate individuals. And that for me is what’s sad about wicked problems: when you consider burning the village because that’s the only way of saving it, perhaps this serves some greater good, but there’s nothing in it for the people about to lose their homes.

I saw a post the other day about an interview by a former CEO of Apple, John Sculley, who lately thinks AI is going to replace programmers. The caveat? Sculley originally came from marketing, and stopped managing much of anything in 1993. Since that time he’s been an investor. This means he never even was an engineer, or in charge of modern engineers in a way that would inform him whether they could be replaced by AI. Yeah, he was a CEO, but I don’t think he knows what he’s talking about.

Sculley was dismissed as Apple CEO 29 years ago, and replaced by Gil Amelio, who was himself dismissed in 1997, with the company essentially dead in the water. The terrified investors called in the notoriously uncouth founding CEO, Steve Jobs. That move played out brilliantly, because Jobs’ core vision of disrupting the way we experience computers in our daily lives was spot-on. Out of a lost cause in 1997, Jobs made Apple what it is today. The potential of Apple was there before Jobs, but neither Sculley nor Amelio could action that, and probably they just did not see that nearly as well as Jobs did.

I saw another post on HackerNews, about Barnes and Noble’s ostensible turnaround, after years of doing badly. That change was attributed to hiring a CEO who is actually an expert in organizing book stores – who was able to achieve that change by stopping book promotion/placement deals with publishers that were toxic to shopping experience, and reforming Barnes and Noble locations as hubs where people who like books and can bond over that. This is a drastic departure from the previous meta-strategy, which included trying to go into eBooks (which is low-margin and requires tech expertise, hence an awful fit for an American corporate), to open restaurants at bookstore locations (which is likewise a low-margin, high-operations-expertise endeavor) and indeed years of ignoring the fact that Barnes and Noble could actually, dunno, just be a great place to buy books.

The hypotheses I’d like to articulate basing on that:

• Vision is rare and calling out visionaries is hard. Corporate survival selects essentially for optics, like being able to make up hypey headlines, visionary types don’t need that much of that facility, hence don’t develop it.
• The only long-term stable situation regarding vision is not having a vision. Accordingly, by far the presumption regarding the the opinions of a corporate manager - no matter how senior in title - should be that they lack vision.
• Accordingly, it is the essential interest of a business’s shareholders that there is actually a source of vision - and just one such source - for the business to survive over time. This is an easy thing to miss when times are good and profits are accumulating, but it becomes unavoidable in bad times, such as what we’re having now.

“Lemons represent failures” is a failed metaphor. Lemons are just fine. You need their juice to make tahini. For many years lemon juice was the only way to avoid getting scurvy in the Arctic. Yeah, they are sour. Sure they’re not a staple food. But lemons are actually cool, and sometimes even vital.

Like with the failure of the lemon-as-failure metaphor, failure of all kinds is a pervasive inclusion in most things we do even on a daily basis. The rest of this piece will figuratively argue the case of lemons as a source of vital nutrients.

The Forest Where I Walk

Since I’m doing pretty well overall (and am deeply thankful for that), some people assume failure is a rare occurrence for me. The truth could not be further; here are some examples at various scales:

• I refactored some APIs a couple of months ago (a positive thing per se), but introduced a bug in one of the code paths and broke production, not once but twice.
• I let at least two projects in the recent couple of years drag on without clear success criteria, which led both to waste time, resources and good will.
• A few jobs back I had made what I still regard as an important engineering innovation, but failed to convince why it was necessary, and why I was awesome for doing it, which set me up for a fairly rough ride.
• Speaking of which, on that occasion I absolutely failed choosing the right job profile, as a result joined the wrong employer and also failed finding a decent executive sponsor.

Inner work: the praxis of failure as an individual

Only a small fraction of all failures are catastrophic. We need to tune the size of failures that we mind, so that we can practice handling them routinely; that way we can often learn a great deal without actually having had to lose a lot; perhaps we could even avoid some of the catastrophic failures. If I had taken more care of the projects going astray, I would have avoided the need to cancel them traumatically.

Failure never ceases, but as we become more experienced, it becomes rarer. Striving to a zero failure rate is misguided. Zero failures means no risk being taken, which is inconsistent with any form of progress. I will continue writing imperfect code, even though I hope to toil less to make it good enough.

Failure is the beginning of a story. Deliberate change may only begin if there’s a way to describe what went wrong. In turn, the description could be analyzed in order to hatch a plan. Conversely, without a failure to begin with, there’s no room for proactive planning. As an example, my ideas about what went wrong with that employer drove me to reconsider my career, overall in a positive direction.

Team retrospection: steering between the rock and the hard place

A lot of people who successfully grow as individuals fail in inducing organizational improvement. I conjecture that the main cause for that is a taboo on publicly examining failure, which stems from the conflation of failure with blame.

Humans tend to place blame as a part of social power dynamics. A failure is a great cause for a display of power – “I am the boss, and to prove my power, I place blame on people who do badly, lowering their status and raising mine”.

Placing blame is an especially destructive force for tech organizations, due to the fact that tech outcomes depend on entire teams rather than individuals. Very few are the cases in which it’s honest to blame just one or two specific persons. The vast majority of failures are caused by multiple factors, and very often managerial decisions, policies and neglect are a major part of failures.

Given the above, the right call is to work out a culture of tracing back troubles and grievances to their causative factors. To maintain the integrity of such culture over time, it needs to apply to everybody on the team, including the leadership, and be focused on things that can be and are likely to be made better.

Unfortunately instead of doing this right thing, a lot of people and organizations overcorrect and go to the extreme of making all consideration of failure a taboo. While indeed disarming the toxicity of blame culture, tropes like “let bygones be bygone” and “water under the bridge”, become a pretext to refuse to stop and learn. This means that failure is going to continue and compound – because unmanaged failures tend to stand out. And trust me, teams notice such things.

Incorporating retrospection to form a better future

While proper retrospection into the past is crucial to understand what needs improvement (without tearing the team apart by blame), such learning can only be meaningful when we use it to make new decisions.

Assume now you are nearing a new decision point. What do you do now? I suggest conducting a short semi-formal session, perhaps even on one’s own or with a few fellow stakeholders:

1. Put together a list of potential failures informed by the learnings so far,
2. If there’s at least one potential failure that is severe and impossible to mitigate, that could be a great flag to change course.
3. Otherwise, use your domain expertise and previous learnings to propose mitigations for the potential failures.
4. Prioritize the mitigations by their relative importance, and in a lean fashion, focus only those items that make the most sense.

If the above process has been done properly and thoroughly and communicated concisely, the result is clarity around what to do. Few things compare to how such clarity instills confidence in the team - for it indicates their professional well being is looked after.

Wrapping up

It is impossible to be attempting anything of substance and not fail at this level or other. However, given the austere climate we’re in now, a preventable failure could be one failure too many.

For me the conversion of failures into better choices is the absolute core of my executive responsibility. Learning the ropes of how to do it in a given team is therefore a vital part of my own professional growth.

I strongly believe that an agenda that’s grounded on the past and is well thought-out to balance the risk and the gain is the way to go, as long as the one rule of the road is kept: there are minor failures and major ones, and some bad enough to get someone fired, but no failure justifies being unkind - either to others or to yourself. Go forth and good luck.

A few days ago, I heard a piece of gossip on my social network. It was from the perspective of a C-level leader who’s conducting a cost reduction plan, as do a lot of people do these days. The leader asked some of his subordinate middle managers to propose and carry out the appropriate reductions within their teams. After a couple of weeks it turned out that what his underlings did was to “reduce” the one remaining employee who actually had the skills to significantly lower operating costs by reconfiguring some parts of the system. To add insult to injury, it turned out that the same employee had proposed multiple times to reduce costs in the preceding months, only to be downplayed by the same middlemen. By the time that the C-level guy has learnt of this matter, that employee had already been driven out. To say that the C-level person was unhappy would be an understatement.

Most people in the industry have heard many such stories over the years, including at Twitter just last week. A trivial way to interpret this situation would be as “those middle managers are all rotten”. I am not satisfied with this interpretation because there are multiple ways a person can become a middle manager, and quite a few of them try to avoid promoting “rot”. Indeed, I believe that there must be a systemic cognitive phenomenon in play that is inherent in the organizational role of middle management. Without further ado, I would like to present my theory of the Positive Preference Pyramid.

The Positive Preference Pyramid

The pyramid is of the three possible preferences that a stakeholder can have with respect to a specific line of work. The preferences are:

• When people who are driven by a vision of how things should be, I call them Visionaries. In the example above, that’s the C-level leader - he has an overall goal of reducing expenditure, but he also insists on doing it at a certain quality bar, by letting a competent specialist guide the process.

• When people are driven by loyalty, I call that situation “being a Loyalist”. In the story above, these are the middle managers who executed on cost reductions. Their focus on maintening of an illusion of control to keep their boss haply eventually caused them to lose awareness of the most basic things going in their own little corner of the organization.

• The remaining people are driven by labor externalities - that is, factors that are not directly connected to the business value of what they do. Examples of externalities are compensation or satisfaction from carrying out tasks even if of little business value. I call these the Laborers. In the example above, it is the one competent employee, the one who ended up being dismissed.

The Pyramid is not (always) about Organizational Hierarchy!

The first example we considered indeed involved a C-level senior leader as the Visionary, the mediocre middle management as the Loyalists and the unlucky expert as the Laborer. However the Pyramid structure often does not coincide with the organizational structure. The positive preferences exhibited by an individual are contextual to the specific line of work at a specific time, and thus may not be aligned to their organizational roles.

I once heard a phrase about an engineer whom I happen to know in person and consider a genius - “the first two CEOs of that company were good - they would call him, hear him out and do as he said”.

Clearly, the leaders - who were organizationally senior to said engineer - were the Loyalists in this case, while he was the Visionary with respect to quite a lot of the technical culture that eventually made that organization successful for all of them. Indeed, if the leaders had tried to be the visionaries on their own, the outcome would have been worse - likely, much worse.

Overdosing on loyalty

First and foremost, we need to observe that in any given area of enterprise, there is very little middle ground between satisfying others’ judgment and exercising one’s own judgment. A consistent preference for following others over time will quickly lead to loss of facilities for doing well on one’s own.

Upton Sinclair once quipped that

It is difficult to get a man to understand something, when his salary depends on him not understanding it.

(quote via the only remotely related but brilliant blog post Strategic Airpower 101).

A hardened upward loyalist’s sense of self worth derives almost exclusively by the assistance they render to their patron. However since their preoccupation means they had lost the edge for doing something useful on their own accord, the only remaining course of action is to ensure nothing negative happens. A loyalist would thus avoid confronting their patron, even if a struggle in the short run would lead to a greater good in the long run.

When bad times come, problems don’t tend to solve themselves. Instead, people who are so inclined use their problem-solving to get the organization in the clear. The tendency to silo and keep quiet sabotages such work. In such circumstances, the practice of upward loyalty is a major force dragging the entire ship down.

Why do loyalists exist?

This brings about the question – why are loyalists even needed in the first place? Several reasons come to mind.

Loyalty keeps the organization together at short term. Assuming basing trust, alignment and competence, projects should be allowed to run on for stretches of days to weeks without second guessing in between. Loyalty, both upward and downward, is the damping mechanism to allow that.

Even when undesirable, loyalty is often good enough when not facing adversity. To avoid immediate failure, most organizations really need to keep keen on only a few things only. As long as these are addressed, the remaining long tail tolerates “making do”. However over long term it is often the case that “the sleep of reason produces monsters”, that is, complacency gives way to unforeseen vulnerabilities.

Agreeing on vision is hard. Vision necessarily stems from shared knowledge. However, getting knowledge across requires the presence of ego on all sides, and ego causes people to clash and disagree. Most experienced leaders counteract that by establishing a culture of managing dissent and selecting their subordinates to fit that culture. Their inexperienced counterparts just bring on loyalists who simply don’t disagree at all.

Sometimes there just isn’t a visionary available. Modern tech management is very operations-heavy; for a struggling visionary, a loyalist underling may be regarded as preferable to not having anyone at all.

Visionaries have less political capital. The focus on vision leaves less opportunities to deal with the personal needs of people around you - which means less of a capacity for the everyday needs and frustrations of people surrounding them, hence less political capital. Loyalists are exactly the opposite and may accordingly prosper while most of the organization does not.

Reflection on the Rao Pyramid

The idea of a three-tiered pyramid describing organizational behavior stereotype is strongly inspired by a shitpost-turned-into-book called The Gervais Principle, Or The Office According to “The Office” by Venkatesh Rao. You can skip to this review by Scott Alexander for a very intelligent reader’s appraisal of Rao’s main ideas.

The model I articulated here is an attempt to develop the same idea in a way that is mindful of motivational psychology, and, frankly - not as misanthropic. For example, Rao’s admittedly idiosyncratic designation of three tiers are “the sociopaths, the clueless and the losers” does little to promote awareness of self or others; the hurtfulness of the labels prevented me from comfortably applying them to any of the people I’d worked with.

Summary: Organizational Evolution and Some ideas for Remediation

One core idea, that in my opinion Rao hit spot-on is the notion that organizational decline over time is manifested in a hypertrophy of “loyalists” (as per the Pyramid) or “the clueless” (as per Rao) at the expense of the other two layers, eventually making the organization too dull to continue existing in a changing world.

One consequence of considering the Positive Preference Pyramid is asking in what lines of work is vision an absolute necessity? In which it is it not?

For a SW “feature factory”, it is absolutely crucial to have a process vision. For a “moonshot organization”, such as SpaceX or the Skunk Works, it may be bringing together a dream team. A product-led organization necessitates a product visionary. Too much vision could be a bad thing - for example, moonshot organizations tend to function poorly if subjected to an excess of red tape.

A different matter to consider is containment. While the introduction of loyalists is unavoidable, the expectations and process around these need to be explicit and obvious. A loyalist should be held as a useful ancillary, but never allowed to reign.

By and large, a lot of people’s reaction to “seeing how the sausage is made” is nihilism, essentially saying that all organizations are inherently flawed and indeed, unfixable. However it is my opinion that organizations may be designed to last for quite a while, even if made from imperfect human beings (being the only kind of human beings in existence) - but of course, to achieve that, at least the organization’s fundamental human vision needs to be in the hands of someone who’s capable of having a vision.

About once a month or so one of the tech forums I subscribe to is rocked by a question similar to the following one:

I am now in my twenties, and am confidently working in the industry. Should I be making a pause now to pursue higher education?

Questions of this kind serve as a validation magnet for both people who have obtained a degree and those who haven’t or couldn’t; accordingly the discussion often gets personal and defensive; my personal highlight is a person whom I’d known (indeed, liked) for 15 years calling me no less than a shill for Big Higher Ed; so much for all those years, I guess.

Is the academic bias in recruiting entirely arbitrary? While cultural settings tend to favor tradition over utility, hiring is a highly resource-consuming activity for a lot of organizations out there. Accordingly it is subject to considerable evolutionary pressure that should reshape it as time goes by. Yet it seems that the academic streak in recruitment is fairly stable - enough to stop and contemplate why.

The Gained Knowledge Hypothesis

The proclaimed goal of education is the transition from ignorance to knowledge. That knowledge could be intended, such as studying some theory that has practical applications, or it could arrive incidentally - such as gaining experience in reading and writing complex ideas.

Alas, it is commonly agreed that both types of skills gained in a degree are actually only very seldom directly useful for the vast majority of jobs out there. Typically employers select employees solely based on their their tradecraft skills, very often in the form attested by being able to design small snippets of code. Tradecraft skills are typically not taught well at schools. In the rarer cases that specific knowledge is required, it is most often about some SW framework like React, which is not taught academically at all.

The Academic Network Hypothesis

A somewhat frequent trope in “pro/con school” discussion is the statement that the school is a great place to meet like-minded friends who’d further your career down the road.

However, it seems that a candidate having a lot of school friends does not benefit an employer much at all.

While in the recruiting phase, academic networks are great for locating and accessing candidates - however that access is shared between multiple recruiters, which actually means network recruits receive more attention and thus need a higher investment per hire.

After an employee is recruited, the network does not affect that employee’s performance per se - and if anything, provides them an easy escape path when things go wrong.

Also, evidently, having any kind of network is not something employers care about. So overall, while I find having a school network immensely useful as an employee, I tend to dismiss its value as an employment signal.

The Contending Resilience and Independence (CRI) Hypothesis

School is the first phase in the lives of most people where most of the motivation for success needs to be intrinsic. Moreover, unlike prior experiences, higher education typically takes smart people to their limits a few times, which is highly enabling of growth - though also humbling and often traumatic. Some periods, especially during undergraduate studies, are like drinking from the water hose - extremely intensive in terms of needing to learn a lot of things at the same time, quickly and with some degree of quality control. Those skills are universally desirable for knowledge workers, and school serves as a signal of possessing them.

A bit of reinforcing evidence for the CRI hypothesis is the employers’ preference for schools that require hard work, regardless of the domain. In Israel, for example:

• The Open University is open to anyone who can pay tuition, requires a high degree of discipline, and offers a relatively hands-off curriculum at medium pedagogical level. It is regarded as top-tier, or almost that.
• Most colleges are open to anyone who can pay a similar tuition, do not require as much discipline, offer a hands-on curriculum at a relatively high pedagogical level. Colleges are regarded as second tier at best.

Caveat: cheating. That’s an easy one - a cheater is only successful in cheating, and they make themselves hard to tell from honest students.

Caveat: not having to go off track, a.k.a. the Learned Fool effect. Most undergrad study tracks these days are highly structured, meaning that study dilemmas are either trivial or ties; this discourages proactive strategizing by the student. The end result may be an individual who has the skills to take up challenges with excellent optics but fails to take up the challenges relevant for his current circumstances.

Caveat: motivational burn-out. A lot of people succeed in contending with a complex academic situation, but do so for the wrong reasons - a typical example is the typically impossible odds of getting into a tenure track position. Thus, a common failure mode of people with an impressive degree is burning out of motivation once their aspiration is discovered to be unattainable.

The Social/Soft Skills Hypothesis

Besides the most exceptional of students, a success of learning is at least partially a success in communicating and socializing. Coursework relies on complex communication, in writing for the assignments and verbally with peers.

Caveat: very little team play: the formal parts of undergrad studies only rarely involve anything resembling meaningful collaboration.

Caveat: loneliness is commonplace. A lot of students report being lonely; not all student friendships are inspiring, supportive and growth-oriented.

Caveat: lack of leadership skills: there are very few experiences in a school that require somebody to lead somebody else; indeed, the closest that comes to it are thesis advisor leadership, which is more often dysfunctional than not, and administration leadership that is designed to be mostly transparent to the students.

The Epistemological Enhancement Hypothesis

One curious aspect of personal growth that happens in academia but few other places is the understanding that the meaning of “knowledge” and indeed “truth” is highly contextual to the area of study. Thus, a good “intuitive proof” (a highly sought after beast in computer science and math) is a wholly different beast from a “formal proof” (which is what gets you a good grade in an exam), and what is considered good “proof” in history necessarily differs from “proof” in economics.

This kind of intellectual maturity is highly impactful for workplace success, especially for the category that requires multi-disciplinary collaboration and simultaneous awareness of tech and business. For me it explains the success of some philosophy graduates in tech leadership positions - even without the best tools, asking the right questions is very often the edge one needs in a world preoccupied with the wrong ones.

Tying it all up

Naturally, different combinations of employers, roles and potential employees will strike different chords out of the above.

The Israeli job market, that I know well, has a very strong tradition of military unit graduates who team up with their former comrades, often without any formal background or specific knowledge for their new organization. It would appear that for them the network is the determining factor - however it is a much stronger, closer kind of a bond than what academia can come up with.

Most recruiting processes I’ve observed, both in Israel and the US, may be seen as highly and specifically tuned to signal about contending, resilience and independence. Indeed, the common preoccupation with the arbitrary riddles in LeetCode as well as asking for candidates’ GitHub is really a way to signal the preference for employees who don’t get lost on their own. Moreover, in a lot of cases, when a hiring process exhibits redundancy, it is when testing for cheating and motivation - which are exactly the soft spots when hiring for CRI.

Circling back to the original question of whether the 20-something year old should go to school – in my opinion it reduces to the question of whether the said person can represent their contending ability to their employer in a different way than by displaying a good school certificate. For a lot of folks out there there there’s really no good alternative – one can’t just “teleport” into having overcome meaningful challenges.

In any case, writing the above helped me articulate what I should be considering when I interview others and get interviewed myself; I hope this may help you as well; as always, I’d be happy to hear your thoughts on the matter.

The paradox

A veteran colleague of mine remarked a couple of days ago poignantly - “you do realize that all foreign corporations that I’ve seen hate the guts of their Israeli branches, don’t you?”

Many of the present agreed. As for me, it sent me to a train of thoughts - is that observation only correct for the corporate-branch crowd? Why do corporates even operate in Israel? Selling to the tiny and difficult local market is obviously not a sufficient answer. And if Israeli divisions of corporates are so obnoxious, how can the much edgier startups get any business at all?

And indeed, let’s ask a more general question – why is the Israeli tech industry so insanely valuable? Is it just some kind of a bubble? I’m no expert on bubbles, but I do know a few things about business value of technology. I would like to share some of the thoughts on this subject.

What major disadvantages do Israeli companies face?

Very little of the following is considered controversial by people familiar with the scene.

It is widely agreed that for most major kinds of businesses, success is determined by being able to sell well in America. However Israeli businesses face a severe handicap doing that. Selling well in a country requires a leadership that’s tuned into the local culture - especially for a highly nuanced and coded culture like the American one. The typical Israeli entrepreneur has simply never learnt to navigate that nuance. (As an example, just consider the email Noam Bardin publically sent when leaving Google - quite the bull in a china shop).

A common response to the above is that Israelis would run the tech while the Americans focus the sales. Tough luck though. The Israeli school of leadership looks down on investment in operations, planning and communications – typically to the point of shooting themselves in the foot. Since building a long-term viable geographically diverse company requires a deliberate and robust backbone made of the above, Israeli leaderships tend to not rise for the challenge, which means they lose and fade away when the going gets harder.

Perhaps then, have the senior management be foreign, and send the fabled Israeli engineers - ostensibly available cheaply at a dime a dozen - be sent in large groups to attack whatever goals the corporate tells them to? Nope. Many American companies successfully follow this playbook in India, but I’ve never seen it succeed in Israel. Good Israeli engineers are rare, expensive and tend to frown upon the kinds of folly that seems to permeate the aether of bigcorp life. Indeed, it is uncommon for Israeli corporate branch offices to host technically complex profit-center products.

As a side note, a rare and welcome exception to the above was Intel – due to a massive fiasco in the US development center, for about a decade Israel became the main location for Intel’s bleeding edge work on CPU architecture design; while Intel is no longer the sole market leader and Israel is not as strong in it, this has sprung a remarkable deeptech industry in high-performance compute design with exciting companies such as NextSilicon and Mellanox.

How do successful Israeli companies survive nevertheless?

The upside of the Israeli contempt for large organizations and whatever it takes to run them is the Israeli die-hard loyalty to small organic teams, and whatever product those small teams can support.

There are several models in which this ability turns out to be valuable - and worth a lot.

Build to relocate - this model means to bootstrap with a geographic pivot in mind - hack together a product with a small team in Israel, win over a niche and then transfer the bulk of the organization to America. This requires careful choreography and a properly calibrated set of incentives early on, but is doable as long as the head is kept level and the hands steady. I consider Sentinel One and Ultima Genomics as a examples of this pattern.

Going for the long-term with deep-tech - going a bit against the splashy zeitgeist of the 2010s, when a company has a deep technology but lacks a well-formed market, the strategy is to keep low, slow and stable until something changes and the technology is suddenly in a great demand. Such are the stories of Mobileye, Vayyar and Flytrex. This attitude requires persevering through years of austerity, or extreme luck being bought out early – one example of the latter being Alooma.

Build a deep-product company - this is similar to the deep-tech model, but here the play is to focus on a single, predetermined and highly fit product idea from the get go. For a product company, success involves growing organically to a medium size. To pull that growth off, the founders need to be professional managers, which is a rare sight in Israel and almost an impossibility given first-timer founders. Companies following this pattern are Lightricks, Gong.io and JoyTunes.

Project delivery space - likely the only way to genuinely “go big” while remaining mostly Israeli. The focus on repetetive, bean-counted, nothing-to-write-home-about projects is toxic to talent elsewhere but is tolerated in Israel. Indeed, a lot of Israelis genuinely like the short-term thrill of integrations and “professional services” dynamics. One category of companies that works in this space is IronSource, which runs an advertisement empire characterized by rapid, iterative integration with multiple partners. Another bigger, older and lower-margin company in this pattern is Amdocs, which focuses on the telecom billing market. In both cases the traditional Israeli weakness in managing large organizations is counteracted by the fragmentation of the development core across multiple somewhat independent projects.

A low-investment corporate branch providing great complementary value - a very frequent and rather successful model for corporate branch offices turns out to be to own important-but-not-core business lines in a larger corporate universe - this pattern leverages the fact that Israeli individual contributors can be highly motivated by the outcomes even where the American only sees an endless career-dulling execution quagmire. This pattern captures a lot of the projects going on in the Israeli MAGMA branches, for example the Microsoft Israel which maintains a security product core in Israel - that’s definitely a great asset for Microsoft, but hardly a cash cow.

Summary

A lot of people have the notion that Israel may just spring up an American-style large corporation that would be a major commercial success. I consider this notion to be fundamentally incompatible with the kinds of successful organizations that Israelis empirically manage to build. However, there’s a myriad of excellent business opportunities in which Israeli teams can win large rewards by playing to their strengths - and I certainly hope to see more of that going forward.

Today’s post is devoted to those who, even when tried by fire, would not stand down.

The Hero

49 years ago to the day, Isaac Brik was a 25-year old studying for the last undergrad year of industrial engineering at the Technion. He was also an armor reservist at the rank of Major, a company commander in the 113th Battalion. As with most IDF reserve units, the understaffed and underequipped battalion was called up from the synagogues to fight what was thereafter called the Yom Kippur war.

On at least two occasions in the next 3 days, the 113th was sent into battle by incompetent superiors who acted rashly and wastefully with lives, resources and - most crucially when it comes to war - initiative. By the time the war was over two and a half weeks later, only a handful of the 200 men on the 113th pre-war roster had not been killed, medically evacuated or held captive by the enemy (the latter group including the batallion’s commanding officer). Brik’s face had been severely burnt in an engagment with Egyptian commando on October 7th, but he chose not to evacuate and fought for the rest of the war.

After the war Brik was decorated for his valor and rose in ranks in the reserves. Several years later he re-joined the military full time, serving both in both line command and staff roles, and rising to the rank of Major General. His last role was that of Head of Personnel Complaints, ending with 2018.

Criticism and Critique

Since leaving that last role, Brik has become a vocal critic, indeed a thorn in the current army leaderhip’s backside. Without going into all details, the core of his criticism is the lack of military preparedness, especially of the reserves - undoubtedly reflecting the dysfunction that Brik paid for literally with his blood. Brik’s criticism is loud, though well-argumented. It is not pretty and totally out of line with top brass’s typical confident nonchalance.

Most wise men would not speak; however, one of the younger generation’s generals went as much as to argue that Brik was “stuck in 1973”, a comment that may be interpreted as an assertion that the failure of 1973 is not going to recur. However, that position, while superficially plausible, implies one the following:

• Either everybody agrees that the failure will not recur (perhaps with the exception of Brik); then there’s really no use for the army as it is now, and it should doubtlessly be reduced and the budget diverted to productive fields such as education.
• At least some people besides Brik think that the failure may recur; in that case, the person delivering the comment should probably talk to them and figure out what to believe, because the consequences of a mistake would be disastrous.

I side with the second line of thought. History may not say much about the future, but it is all we’ve got, and if history says anything at all is that failures persist until addressed.

The Rhyme of Time: war in 1973 and tech in 2022

The Yom Kippur War was culturally a consequence of a great cultural upheaval in many areas of Israeli public life, reaching its peak by the great military victory in the Six-Days War of June 1967.

While war is about lives and tech “only” about livelhoods, there are several parallels to be made:

One crucial angle is the “builders” vs “followers” one. The success of 1967 was a result of 15 preceding years of hard-pressed “builders” who wrote the playbook for a successful war, but who themselves were capable of revising it. That generation had almost entirely moved on by 1973, replaced by “competent followers” – however as such, they failed to grasp that they are in a new era with new rules.

One of the aspects of the 1967 victory was that it cemented the foundational trust into an arbitrary, flawed and unstable superstructure - that proceeded to crack when the shock came. The same applies to multiple organizations established in a period of plenty, who cemented otherwise resonable business cases with unsustainable (and often times, unnecessary) business practices - and now the entire thing is cracked.

And making a full circle to General Brik, the most intolerable thing for many has been the hubris of success - the insistence on the wrong solutions to the wrong problems asserted by the wrong people. For sure, “prophecy has been taken from prophets and given to fools and children”, but being a fool or child is a neceessary condition rather than a sufficient one, and indeed “you don’t need a weatherman to know which way the wind blows”.

Having just passed the Yom Kippur of 2022, I wish all my readers that they may have an opportunity to improve their own judgement - and that they choose to do so.