The Cropper’s Skill

Think about what made croppers valuable. They endured years of painful training to develop a very specific physical skill. The skill itself wasn’t the point. The point was to produce a well-finished piece of cloth. But because the skill was hard to acquire, it became a proxy for the outcome. If you had the saddle, you could finish cloth. If you could finish cloth, you were valuable.

It’s unlikely that anybody actually wanted to develop a giant callus on their forearm. The callus was a side effect of the real work. But over time, the callus became the credential. You could tell a cropper by his saddle.

The LeetCoder’s Skill

Now think about what makes a LeetCode grinder valuable, or rather, what is supposed to make them valuable. They spend months memorizing algorithms and data structures, practicing problems until they can solve them under time pressure. The skill itself isn’t the point: the point is to identify people who can write good software. But because the skill is hard to acquire and (loosely) related to writing software, it became a proxy for the outcome.

It’s unlikely that anybody actually wants to memorize the algorithm for finding the longest palindromic substring. That knowledge is a side effect of what companies are ostensibly looking for: the ability to solve real engineering problems. But over time, the LeetCode performance became the credential. You can tell a grinder by their leetcode performance.

I’ve written before about how to separate LeetCode grinders from people who can actually code. Even back in 2022, the distinction mattered. Grinders could ace a standard interview while being mediocre at the actual job. The proxy had already diverged from the outcome it was supposed to measure.

Croppers Were More Useful

Here’s the thing, though. Croppers, for all the absurdity of using a callus as a credential, were actually doing the real work when cropping. The saddle was a byproduct of genuinely finishing cloth. A cropper’s skill and the output were directly connected. A great cropper produced great cloth, and a bad cropper produced bad cloth. The credential was imperfect, but it was at least correlated with the real thing.

LeetCode grinding doesn’t necessarily have that property. The ability to implement a red-black tree on a whiteboard or on a notepad-like interface in a video interview has, at best, a weak correlation with the ability to design a reliable and useful system. The proxy was always more disconnected from the outcome than the cropper’s saddle ever was.

So even before AI came along, LeetCoders were already a worse version of croppers: people who developed a painful, specialized skill that served mainly as a credential, except the credential was less connected to the actual work. Hopefully, when hiring a leetcoder, you were getting a good software engineer, and in many cases you were. But there was a lot of “false negatives”: people who could write great software but didn’t do well under pressure, or did not have a lot of time to memorize algorithms while developing a real system and raising a family. There were also a lot of “false positives”: people who could grind LeetCode but weren’t great engineers.

Rage Against the Machine

And now, of course, there’s a machine that does it all.

AI can easily solve LeetCode problems. Not just the easy ones, it also handles hard problems reliably and quickly. The entire ritual of memorizing algorithms, practicing under time pressure, and performing in a 45-minute interview can now be done by a system that costs a few cents per query. The machine doesn’t need months of preparation, doesn’t get nervous, and doesn’t forget the algorithm after the interview is over.

In addition, in the last couple of months, creating software with AI agents have felt different. Agents are tackling more complex problems, learning from the existing codebase, and writing code that is very good. The value of being who memorized the syntax of a particular programming language has declined. And since AI models have memorized all Leetcode problems, the value of being someone who also memorized them has also gone down.

What’s Left

When Mellor and the croppers saw the machines, they knew their skill was becoming worthless. John Booth, the young outsider, tried to tell them that the machine could be “man’s chief blessing instead of his curse,” if the benefits were fairly shared. But that was cold comfort to the workers, and in the end, even Booth couldn’t hold to his own argument. He joined the Luddites and died in the raid on Rawfolds Mill, at nineteen.

The croppers’ trade didn’t adapt. It simply disappeared. But the cloth industry didn’t disappear with it. It grew, and the people who thrived in it were the ones who understood cloth, not the ones who had the biggest calluses. The skill that mattered was never the shearing itself. It was creating cloth products that users wanted and needed.

The same will be true for software engineers. LeetCode performance is the saddle: a painful credential that was always an imperfect proxy for the real thing. What will matter is what always mattered: judgment, taste, and knowing what good software looks like. Now you won’t need to code everything yourself and memorize syntax. You can simply direct AI agents that write the code for you.

The saddle was never the point. The cloth was.

The problem

I love to travel, and as an applied scientist, I get to travel a lot for work, visiting partners in the field, and presenting the work of the [AI for Good Lab]. In addition, I also love to travel for leisure.

Working with AI (training models or even doing local inference) meant having a “mobile workstation,” a very heavy laptop that can train models locally. For many years I used the Lenovo P-series, with my latest being a Lenovo P15 Gen2 that served me well since 2021.

The problem: it weighs 6 lbs (without counting the gigantic power brick) and it goes through its battery in about a couple of hours. Charging it is tricky, too - it does not charge from USB, and its power brick draws so much power that it prevents charging in most planes.

So I started looking for a lighter computer. Given my long-time loyalty to Lenovo, I started there, but I was quickly underwhelmed by the selection. The smaller and lighter Lenovo P14s felt underpowered, with Ada GPUs with only 4 Gb of VRAM, or Blackwells with 6 Gb of VRAM, which restrict the size of models I can use locally.

So, after many years of being a Lenovo-only household, I looked outside, and I found two laptops that I liked.

My requirements

• Large GPU VRAM for local AI work. 12-16 GB is the difference between “this is a fun demo” and “I can actually do something useful on the road.”

• A truly portable chassis. I need a decent battery, and also a laptop that fits in the tray of an economy seat.

• Practical charging. I wanted something that could top up over USB-C for “normal work” days (email, notebooks, writing, light coding), even if full GPU performance still needs the big brick. Also, it’s helpful to have some way to charge while I rest during a flight.

Both the ASUS ROG Zephyrus G14 and the Razer Blade 14 fit all those requirements. (These are affiliate links.)

• ASUS Zephyrugs G14 product page
• Razer Blade 14 product page

Review comparisons consistently show that the Zephyrus G14 delivers excellent real‑world performance for its price, great battery life, and an OLED display that delivers deep contrast and rich colors, which benefits both content creation and media consumption.

In contrast, the Razer Blade 14 leans more heavily into premium design, which gives it a slightly better thermal envelope for the GPU (increasing the performance a bit), and slightly higher refresh‑rate displays. However, this is currently limited to a RTX 5070 with 12 Gb of VRAM for $2,300. The same configuration for a Zephyrus G14 is $1,999. I was able to find a Zephyrus G14 with a RTX 5080 16 Gb VRAM for $2,550, $250 more than the Blade with a 12 Gb 5070, and I got it.

Early “real world” notes

A few things became clear almost immediately after switching away from the old Lenovo.

• USB-C charging is an incredible quality-of-life upgrade. These days, everything charges off USB-C, so even if the computer is charging a bit slower than the full power brick (200W for the power brick vs 100W for the USB-C), it still charges properly overnight. Better yet, I can charge it from my [power bank]

• It fits an economy class tray with space for a drink (the laptop has to hang a little out)

• It showed 4h30 hours of battery in a flight without having to adjust power configurations or dial down the OLED brightness (no GPU use). My flight was shorter, so I only used 2h.

• It’s sufficiently powerful when connected to the power brick: the RTX 5080 seems to be equivalent to a V100, enough to run a lot of real-world workloads.

Closing thought

I honestly expected Lenovo to have a better offer, and that would have saved me given the number of power bricks and docking stations that my family and I accumulated over the years. But 4 Gb VRAM is insuficcient these days for a small form-factor, and the larger form-factors are impractical. I’m glad Razer and ASUS have ultraportable powerful computers available.

Warming up to information

In February 2023, the Missouri Tigers basketball team (then 17–5) were heading to play the Mississippi State Bulldogs. On paper, the Tigers were the favorites, and the betting markets reflected that. However, the flight taking the Tigers’ players got delayed due to mechanical issues, which meant that they would not arrive in time to perform their warm-ups.

You likely don’t know this, but teams that don’t warm up usually lose their games. Of particular importance, at the time of this game, teams that didn’t warm up were 0-13 against covering their spreads. A spread is the margin by which the favored team is expected to win. Covering the spread means meeting or exceeding that margin if you’re the favorite, or staying within that margin (or winning) if you’re the underdog. For example, if a team is favored by 5 points and wins by 7, they have covered the spread. If they’re expected to lose by 5 points and lose by only 2 (or win), they have covered the spread.

Quickly after the flight delay announcement, large bets started pouring in for the underdog Mississippi State. Sportsbooks took notice and opened an investigation. The investigating company contacted the Missouri athletic director and found out that they were running late. The time between the announcement, the surge of bets, the investigation, and the Missouri athletic director receiving the call from the investigators? Six minutes.

Mississippi State went on to win 63–52, covering both the first-half and full-game point spreads and confirming once again the importance of warm-ups. This travel hiccup turned the odds on their head—and turned out to be a winning bet for people who had the information early.

A typical Azure Web Apps workflow

Azure Web Apps know how to work with Python repositories that contain a requirements.txt file. If you’re building a Python app with Django, Flask or FastAPI and deploying it to an Azure Web App, you probably have a GitHub Actions workflow YAML that has a build step that looks like this, under the .github/workflows directory of your repository:

  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python version
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Create and start virtual environment
        run: |
          python -m venv venv
          source venv/bin/activate
      
      - name: Install dependencies
        run: pip install -r requirements.txt
        
      - name: Zip artifact for deployment
        run: zip release.zip ./* -r

As you can see, this workflow checks out the repository, sets up Python 3.12, creates a virtual environment, installs the dependencies as listed in requirements.txt, zips the artifact and uploads it to the GitHub Actions runner.

If you have been using uv, you know that you don’t need to manually create a virtual environment, and you don’t need to list your dependencies in a requirements.txt file. You can do this locally with uv by exporting the dependencies with uv pip compile pyproject.toml -o requirements.txt, but that would add an extra step before checking in your code. Instead, we’re going to do this directly in the GitHub Actions runner.

Replacing the virtual environment and requirements.txt steps with uv

In order to use uv during the Azure build, you need to install uv in your GitHub Actions runner. You can delete the Create and start virtual environment and add the following code to install uv:

      - name: Install uv
        uses: astral-sh/setup-uv@v5

This will install uv in the GitHub Actions runner. The next step is to generate the requirements file. You can do this with the following command:

      - name: Generate requirements.txt
        run: uv pip compile pyproject.toml -o requirements.txt

Interestingly, deleting the Install dependencies step seems to have no effect at this point. If you need the packages during the build step, for example to perform tests, you shouldn’t delete them, but apparently the deploy step of the GitHub Actions workflow will install the dependencies from the requirements.txt file even if you don’t install them explicitly in the build step.

Just do it!

I have been using uv for the past few months, and I can tell that my producitivity improved. It’s easier to manage packages, to maintain installations, to install pytorch, and a lot easier to perform tests using multiple versions of Python using uv.

The only thing that was holding me back on using uv more was the lack of native integration with Azure Web Apps, but now that I have a workaround, I can’t see myself going back to the old way of doing things.

Range

My Kia EV9 is the Land long-range AWD trim, with a 99.8 kWh battery. The listed range is about 290 miles, but that assumes a mix of city and highway driving and some idealized conditions. This listed value is somewhat misleading in the real world for many reasons, one of which is that you don’t want to get to your destination with 0% battery: keeping a reserve charge is important to avoid being stranded due to unexpected delays, detours, or non-functional chargers. Therefore, you should count that the best you can consume on a trip is about 90%. Even if you were to take a 10% discount and use a range of 261 miles, you may find that a lot of the mental shortcuts that you could use for internal combustion engine (ICE) vehicles (gas-powered cars) don’t work well for electric vehicles (EVs).

One mental shortcut is that for long highway trips, ICE vehicles tend to go farther than their listed range. The listed range assumes some proportion of highway driving and some proportion of city driving. In ICE vehicles, the highway consumption is lower, so an ICE vehicle with a listed range of 261 miles will probably go farther than that. The opposite is true for EVs: at high speeds, consumption is higher than average, so the actual highway travel range of the vehicle is likely lower.

From my own measurements, my EV9 can do about 3.5 miles per kWh in the city (which would mean a 90% range of 315 miles) or about 2.7 miles driving at around 65 mph in a Houston highway (which would mean a 90% range of 253 miles). My planned interstate trip would have me driving at higher speeds, and therefore, my range would be even lower. I figured I’d need to charge twice unless I wanted to arrive almost empty.

Chargers and Planners

These days, most hotels in the South of the United States don’t have EV chargers (either slow or fast), and the city of New Orleans also doesn’t have many public chargers. New Orleans is very walkable, but we wanted to do some things that required driving. Luckily, one of the places we wanted to drive to (Audubon Zoo) had a free slow charger that charges at 6 kWh per hour, so we could leave the car there for a few hours and get a little bit of charge. I planned for a 50% charge on arrival and assumed I’d be able to charge 30% at the Audubon Zoo on the day we visited it.

Although I planned the whole route at home, I still had two concerns: one is that I hadn’t yet driven the Kia EV9 on a long trip, so I didn’t know what mileage I should use. The higher speeds I expected at the trip might lower my mileage substantially, but I didn’t know exactly by how much. The other is that I didn’t know which chargers would be working (a common problem these days), and whether they would be busy or not. With fast chargers spread far apart, a non-working charger could mean a long detour for which I might not even have enough charge to get to the next one.

For some vehicles, Apple Maps can help: the vehicle provides real time information to Apple Maps using CarPlay. However, the Kia EV9 does not support CarPlay yet, so I first tried using its built-in route planner, but it was frequently planning a route that used chargers that I suspected wouldn’t be working. To confirm my suspicions, I used the PlugShare app and found out that several chargers suggested by Kia were closed at night, had restrictions like “Rivian owners only,” or were out of service.

Searching online, I got the recommendation to use a range calculator called A Better Route Planner, usually abbreviated to ABRP. I installed it, and it asked me about my car model and trim and whether I had a NACS (Tesla) to CCS (legacy fast charger) adapter. I have an adapter that I use in my other EV (a Ford F-150), so said yes, and it told me that I would need two stops to get to New Orleans with 50% charge. It planned a route with an Electrify America (CCS) charger in Sulphur, LA, and a Tesla (NACS) charger in Baton Rouge, LA. The total distance was 400 miles, and the total time was about 7 hours, including charging stops. I subscribed to the premium version of the app, which allows you to see real-time data about your car and adjust for climate and traffic, which would help me adjust the route as the app gathered real-time information about actual mileage, traffic and temperature.

A Worse Route Planner

As we set out, I started measuring the real-time consumption. At about 75 mph, the actual consumption was about 2.4 miles per kWh, but at 80 mph, it was about 2.1 miles per kWh. At higher speeds, a lot of the advantages that make EVs energy efficient disappear. Even driving fast, we made it to Sulphur and quickly charged at Electrify America, reaching charging speeds of 200kW, which got us to 80% in about 20 minutes. Most fast chargers start charging slower when the battery state-of-charge (SoC) reaches 80%, so charging from 20% to 80% may take 20 minutes and charging from 80% to 90% may take an additional 20 minutes. Since the next planned stop in Baton Rouge was also a fast charger, to save trip time, I didn’t “top off” the car to 90% or 100%.

I got to Baton Rouge to a bad surprise. I did not know then, but even though the app said I could charge at the Tesla supercharger, it turns out that I can’t, even with the adapter. The reason is that Kia vehicles from charging in the Tesla superchargers until Jan 15th, 2025, even if you bring your own adapter. They can only charge in Tesla “magic chargers”, a Tesla charger that includes a Tesla-owned attached adapter, but these chargers are not available in all locations.

I then had two options: a detour to Electrify America in Hammond, LA, which meant a few extra miles, some extra time and reaching the charger with the battery at about 5% or keep going to a new group of Shell Recharge fast chargers at Tangers, LA, which was a lot closer to New Orleans. The problem: if that didn’t work, we would be far away from other chargers. I decided to take the risk.

Shell (doesn’t) Recharge

As it’s sometimes the case, the risk didn’t pay off. Upon arriving at the Shell Recharge station, I was required to install the Shell Recharge app to activate the charger. The process was cumbersome, involving a $75 hold on my credit card for each attempt. Despite multiple tries and remote support assistance, most chargers failed to work, and the one that did could only charge at the unusably slow speed of 1 kW, meaning that it would take about 75 hours to charge the car. In total, I got a hold of $750 on my card (released after a few days) and lost about one hour.

At that point, both Hammond and New Orleans were out of reach, and the teenagers in my car were hungry and cranky, so instead of having a Cajun dinner in New Orleans, we ended up going to the always reliable Waffle House, and I let my car charge for a couple hours in a slow ChargePoint charger, which gave me enough range to get somewhere. Bellies full, we decided that we would go to Hammond, use the reliable Electrify America and arrive with a charged car in New Orleans.

New Orleans chargers and the trip back

Although New Orleans downtown doesn’t have a lot of fast public chargers, the slow chargers I found were free, so I parked on them and recharged the car as we walked around at Audubon Zoo, or downtown while consuming beignets.

Lessons learned, I deleted ABRP, manually planned a route back using Electrify America chargers, and decided that my next trip with the Kia EV9 would be only after Jan 15th, 2025, when my Kia will have access to a much larger charging network.

Final thoughts

EVs are new, and we are having the usual growth pains for something that requires that much infrastructure. I was enormously happy that I could do my whole trip using my EV. At home, my EV is charged with solar power, so this means that the carbon signature of my trip was a lot lower than if I had driven a gas-powered car.

Even though the adapter mishap and the problems with broken chargers were stressful, I am really happy that I chose to exclusively use EVs. The significantly lower carbon emissions compared to gas-powered cars and the cost and emissions savings from charging with solar power at home reaffirmed my decision. These benefits far outweighed the challenges I faced during this trip. They worked well enough for what I needed, and I know how to make the next trip better and perhaps even stress-free: I’ll just wait a couple of weeks until both my EVs can use all fast-charging networks.

It’s Just Adding One Word at a Time

The main idea behind LLMs is that they generate text one word at a time. The model is trained to predict the next word in a sequence of words, given the previous words. To understand how a model can choose the words, you can think that it uses a list of words and the probabilities that they will come next. For example, for a phrase like “The best thing about AI is its ability to”, the list of probabilities for the next word might look like this:

These probabilities come from the corpus of text that the model was trained on. It is natural to think that the model should always choose the word with the highest probability, but if we always do that, we typically get a very boring output. Therefore, we allow the model to choose a word with a lower probability, which makes the output more interesting. The temperature parameter controls how “creative” the model is. The higher the temperature (a real number that goes from 0 to 1), the more likely the model is to choose lower-ranked words.

The model repeats this process for each word in the sequence, generating text one word at a time. For example, if the model generates the word “learn” as the next word, it then uses the phrase “The best thing about AI is its ability to learn” as the context for predicting the next word. This process continues until the model generates the desired amount of text.

Here are three completions I ran in ChatGPT for “The best thing about AI is its ability to”:

• learn, adapt, and solve complex problems, enhancing human potential and transforming industries with innovative solutions.
• analyze vast amounts of data, uncover patterns, and provide insights that would otherwise be impossible for humans to achieve on their own.
• continuously evolve and tackle diverse challenges, enabling more efficient, personalized, and creative solutions across countless domains.

Where the probabilities come from

Although the probabilities do come from the corpus of text that the model was trained on, there’s no giant table with all possible combinations of words and their probabilities.

The reason is that the table would be too big, with a number of entries larger than all the atoms in the universe. Instead of having a table of probabilities, LLMs create a model based on a deep neural network. The model has a fixed size, called the number of parameters. Parameters are numbers that the model learns as it trains. Neural networks work with numbers, so the words are converted to numbers using a technique called embeddings. Embeddings are numerical representations of words done in such a way that similar words have similar embeddings, placing them close to each other. The image below shows embeddings for some words in a 2D space.

The model shown a sequence of words with a word obscured. It then tries to predict the obscured word’s embedding. The input is converted to numbers, and the numbers are modified (through multiplications and sums) by the parameters of the model. The training program knows the right answer, and if the model gets close to the correct answer, the parameters are reinforced. If the model is far from the correct answer, the parameters are adjusted. This process is repeated trillions of times, with the model getting better and better at predicting the next embedding in a sequence. Once the model is good enough, you can stop training it and start using it to generate text.

It can’t be that simple

At this point, you may be thinking — it can’t be that simple. And you are right. There are a lot of other details that I intentionally skipped to make this post as simple as possible for a beginner. The descriptions above apply to earlier versions of large language models such as GPT-2, back when you could run such a model on your own computer. These earlier models created interesting human-like outputs but they didn’t seem like magic. Today, models are trained on a lot more data, and require a lot more computational power to train. The original GPT-2 had 1.5 billion parameters. The recently released Llama 3.3 has 70 billion parameters.

Another change from these original large language models was the introduction of Reinforcement Learning from Human Feedback (RLHF) training, a way to train a model to generate text that is more likely to be useful to humans. This was done by showing humans the text generated by an LLM and asking them to rate it. The generated text and the ratings are then used to retrain the model to generate better text. This can be done in an external model that interacts with the LLM or you can simply allow the model to update itself. This idea is generally known as fine-tuning and an early version that performed extremely well was implemented in InstructGPT.

Where to go from here

I hope this post helps you understand a little better how LLMs work. If you want to go deeper, I recommend reading the original article by Stephen Wolfram.

Quarto and Bluesky comments

I use Quarto to write my blog. It’s free, allows me to write in Markdown and write customizations in Python, and I can host my blog for free on GitHub or Netlify (I chose Netlify). One perpetual annoyance for bloggers is to enable comments. People who use the popular WordPress platform can use a plugin, but there’s a lot of spam.

Recently, Garrick Aden-Buie created a plugin that allows you do combine two actions: promoting your blog post on Bluesky and using Bluesky to manage comments.

His full solution is on his website, and it’s very simple to use, so I’ll just describe it here.

Enabling comments and a chicken and egg problem

The first step is to install the Quarto extension. To do so, assuming you have Quarto installed, you can simply run the following command:

quarto add gadenbuie/quarto-bluesky-comments

You can publish your posts normally. To add the Bluesky comment functionality, you first need to share the post to Bluesky, to get the post (Bluesky posts are called skeets) ID. For example, I wrote my post about Birdbuddy and shared it to Bluesky in the following skeet: https://bsky.app/profile/lucasmeyer.bsky.social/post/3lbxfiyp7jc2a.

Once you have the post/skeet URL, you have to go back to your blog post, ad add the following code at the end of the post and republish it (this is the chicken and egg problem):

{{< bluesky-comments skeet-url >}}

To make my blog post look pretty, I added a heading, so the final code for the Bluesky post would look like this:

## Comments {.unlisted}

{{< bluesky-comments https://bsky.app/profile/lucasmeyer.bsky.social/post/3lbxfiyp7jc2a >}}

Now comments to the Bluesky post appear as comments on the blog.

Gathering the postcards

The first step is to gather the postcards. Birdbuddy doesn’t really have a public API, but it uses GraphQL, and people have reverse-engineered the interface and created a Python library called pybirdbuddy that can download the postcards from the Bird Buddy app.

The code below shows how to get the pictures from the postcards that were collected in the last 24 hours:

    bb = BirdBuddy(BIRD_BUDDY_USER, BIRD_BUDDY_PASSWORD)

    # get the last day
    since = datetime.datetime.now() - datetime.timedelta(hours=24)    
    postcards = await bb.refresh_feed(since=since)

    for card in postcards:
        if card.get('__typename') == "FeedItemNewPostcard":
            sighting = await bb.sighting_from_postcard(card.get('id')) 
            species = sighting.report.sightings[0].species.name

            media_items = [{'id': item['id'], 
                            'date_created': item['createdAt'], 
                            'species': species,
                            'media_type': item['__typename'], 
                            'image_url': item['contentUrl']} 
                            for item in sigthing.medias 
                            if item['__typename'] == 'MediaImage']

The downside is that this will download all the pictures, and many of the pictures are not very good. I want to filter them out automatically.

Is this picture any good?

To filter the pictures, I use the GPT-4o model with a simple prompt. Here’s the Python code that I use:

def good_bird(image_url):

    openai_client = OpenAI(api_key=app_config.OPENAI_API_KEY)
    prompt = f"I want to post cute and interesting images of birds to social media.\n"
    prompt += "Is this image such a picture? Reply 'Yes' if it is good, otherwise 'No'."

    response = openai_client.chat.completions.create(
        model="gpt-4o", 
        messages=[
            {"role": "system", "content": "You are a photography critic and social media content creator"},
            {"role": "user", "content": [{"type": "text", "text": prompt}, 
            {"type": "image_url", "image_url": {"url": image_url}}]}],)

    return 'yes' in response.choices[0].message.content.lower()

Running the list of media_items through the good_bird function, I can filter out the pictures that are not good enough to post.

I also use GPT-4o to generate a caption:

def caption(species, image_url):

    openai_client = OpenAI(api_key=app_config.OPENAI_API_KEY)
    
    prompt = f"Generate a caption for this {species} that was captured on a bird feeder camera.\n"
    prompt += f"Do not assume the bird's gender.\n"
    prompt += f"The caption will be used in a social media post and should be less than 200 characters.\n"
    prompt += f"The caption should be suitable for a professional brand, although it can be funny.\n"
    prompt += f"Do not use emojis or hashtags.  Do not ask for engagement. Do not ask questions.\n"

    response = openai_client.chat.completions.create(
        model="gpt-4o", 
        messages=[
            {"role": "system", "content": "You are a photographer and social media content creator"},
            {"role": "user", "content": [{"type": "text", "text": prompt}, 
            {"type": "image_url", "image_url": {"url": image_url}}]}],)

    return response.choices[0].message.content

Now each image has a caption and a flag indicating if it’s good enough to post. In my app, I’m saving all the pictures to a database and posting a few of them every day, but you could post it directly from the code above.

Posting to Threads and Bluesky

To post the pictures, I use the Threads and Bluesky APIs.

The code below shows how to post a picture to Threads. Posting to Threads requires two api calls - one to an endpoint called threads to create the post and upload the assets another to an endpoint called publish to actually publish the post. You need a token and a user id to post to Threads, and you can obtain both from the Meta Developers website. Please note that the code below is a simplified version of the actual code, and you should handle errors and exceptions properly.

def post_to_threads(message, image):
    payload = {
        "access_token": THREADS_TOKEN,
        "text": message,
        "media_type": "IMAGE",
        "image_url": image
    }

    post_url = f"https://graph.threads.net/v1.0/{THREADS_USER_ID}/threads/"

    response = requests.post(post_url, json=payload)

    if response.status_code == 200:
        # get response id
        response_json = response.json()
        creation_id = response_json["id"]
    
        publish_payload = {
            "access_token": THREADS_TOKEN,
            "creation_id": creation_id
        }

        publish_url = f"https://graph.threads.net/v1.0/{app_config.THREADS_USER_ID}/threads_publish/"

    response = requests.post(publish_url, json=publish_payload)

Posting to Bluesky is similar, but a lot easier. You must have created an app password, your account password can’t be used with the API. The code below shows how to post to Bluesky. In order to post to Bluesky, you should install the package atproto, which allows you to import a Bluesky Client class.

def post_to_bluesky(message, image, image_alt_text):
    client = Client()
    client.login(BSKY_USER, BSKY_APP_PWD)

    image_data = requests.get(image).content
    client.send_image(message, image=image_data, image_alt=image_alt_text)

Want to see the posts?

You can see my Birdbuddy posts on Bluesky. They are all tagged with #birds. I hope you enjoy them!

Buying your own Birdbuddy

Right now, I own two Birdbuddies, the original version in my backyard and a Pro version in my front yard, and I’m likely to buy the hummingbird one soon.

If you want to buy your own Birdbuddy, you can find it on Amazon.

Croppers

A central figure of the Luddite movement was the “cropper”, a skilled worker who used large shears to finish cloth pieces. The shears were heavy (40 lbs) and unwieldly, and using them well was tiring and painful, requiring a lot of training. Over time, croppers developed a large callus on their forearm (called a saddle), and it was said at the time that “you could tell a cropper by his saddle”.

A good finish could mean the difference between a piece of cloth that would sell for a high price and one that would need to be thrown away. Given their importance, the training requirements, and the physical toll of the job, croppers were well-paid, high-status workers. But then, in the early 1800s, some machines were invented that could do the job of the croppers as well as the croppers themselves, but also faster and cheaper.

The conversation below was reported by the nineteenth century historian Frank Peel, who recorded oral stories handed down over generations in his book The Rising of the Luddites.

The Conversation

In a cropping shop, a group of men are talking after work. Most of the men are croppers, and recently started forming gangs to smash the new machines. One of men is John Booth, a nineteen-year-old apprentice saddle maker, the son of a clergyman with the Church of England. Since he is not a cropper, he’s not directly affected by the new machines. Not yet, anyway.

“Would it not be better,” asks Booth, “to reason with them rather than infuriate them by destroying their machines?”

“Reason with them!?,” says one of the croppers, “We might as well reason with a stone!”

The cropper is named George Mellor, and he’s the local leader of a masked marauding gang that destroys the machines that are replacing workers. Mellor is still working for now, but he knows that he can’t compete with the machines. He and his fellow croppers can see that the skill they’ve spent years perfecting is soon to be rendered worthless.

Booth clearly thought this out, and says that “being a cropper is hard work, and painful too, until you develop that hard, calloused skin on the wrist. But have you seen those machines in action? You just have to set up the cloth and keep an eye on them. They take away all of that hard, painful work. Seen like that, the machinery is a thing of beauty.”

Booth continues, “I quite agree with you about the problem, and respect the harm your livelihood suffers from the machinery. But such a machine might be man’s chief blessing instead of his curse. You can’t say the machine itself is evil, that’s absurd! No, the problem is that the mill owner gets all the benefits. If society was differently constituted, if those benefits were fairly shared out…”

Mellor interrupts him. “If, if, if…” he says. “What’s the use of such sermons as thine to starving men?”

England is at war with Napoleon’s France, and the war has disrupted trade. Nobody’s hiring, and the price of food has shot up. Another cropper joins in the conversation. He tells how he went to see a former workmate, a cropper who lost his job. He can’t find work. The man’s been struggling to afford to eat, and now his wife has died. “There she lay on the bed, poor thing, skin and bone.”

It’s hard for John Booth to argue with that. He is sure that the machines would be a blessing if society were differently constituted. On the other hand, he also knows that his cropper friends are right. There’s no chance of reorganizing society at present, and there’s no comfort in imagining a different societal organization when you are starving to death.

“What can I say?” asks Booth.

“Say you’ll join us,” says Mellor.

The Luddites and General Ludd

The gangs were not a single, unified group. They were a loose collection of people who were angry at the new machines and the mill owners who were using them.

There was a local story that a young weaving apprentice named Ned Ludd smashed two machines in a fit of rage. When gangs started smashing machines, the story goes, instead of snitching on them, the other workers would say that “Ned Ludd did it”. Over time, the gangs incorporated that legend, and said that they were acting on behalf of General Ludd. Incidentally, General Ludd was supposed to live in Sherwood Forest, same as Robin Hood. Before striking, the gangs would usually send letters to the mill owners, ordering them to stop using the machines otherwise they would be smashed. The letters were signed “General Ludd”. The followers started to be called Luddites.

For a while, the Luddites were successful. Mill owners were concerned, and some stopped using machines. However, a few mill owners held out. One of the holdouts was William Cartwright, the owner of Rawfolds Mill. On April 11th, 1812, more than than one hundred Luddites gathered at the Dumb Steeple, still a landmark near Mirfield, in Yorkshire. At the dead of night, they marched on Rawfolds Mill. But Cartwright was ready for them. The mill was fortified and soldiers were stationed there. The Luddites were beaten off and two of them were killed.

One the dead was the nineteen-year old John Booth, the young apprentice saddle maker who had been trying to reason with the croppers. Legend says that before Booth died, a reverend tried to extract from him the name of the local leader of the Luddites under the seal of the confessional.

Booth asked the reverend, “Can you keep a secret?” The reverend said that he could. “So can I,” said Booth, and died.

The Mill Owner’s Problem

It’s easy to see the mill owners as the villains of the story. They were the ones who were buying and using the machines that were putting the croppers out of work. They were the ones that put soldiers to fight against the common man. But the mill owners had a problem too. Machines were making the production of cloth much cheaper. If they didn’t use the machines, they would be outcompeted by other mill owners who did.

Even if the Luddites were successful across the whole kingdom of England, a different country could organize itself to start using the machines. If that happened, the mill owners would still go out of business, and all of their workers would be out of work, including the croppers.

The Luddites and the AI Revolution

John Booth’s argument was on point. Centuries later, the economists Daron Acemoglu and Simon Johnson wrote the book Power and Progress, in which the central argument is not very different from the one in John Booth’s conversation. “Progress depends on the choices we make about technology. New ways of organizing production and communication can either serve the narrow interests of an elite or become the foundation for widespread prosperity”.

In the book, they provide several other examples spanning over a thousand years. The examples include resource allocations that did not work, like the agricultural innovations in the middle ages that led to larger and more opulent cathedrals. They also include allocations that did work, like the assembly automation in the 20th century that led to the rise of the middle class and the increased prosperity of the United States post World War II.

Now, with AI, we again have a technology that can do the job of many workers faster and cheaper. Will we, as a society, do better this time?

What the future holds

It seems inevitable that AI will be incorporated more and more in the workplace and in day-to-day life, to the extent that it may be pointless to try to stop it. Several countries already have defined strategies to incorporate AI in their economies, so even if a fight against AI is successful in one country, that victory may translate into a much bigger loss. Avoiding AI is unlikely to be a long-term winning strategy.

When people ask me about AI, I frequently tell them to start using it in their day-to-day, to understand its limitations and its potential. I suggest that the get a suscription like OpenAI’s ChatGPT or Bing Copilot and attempt to use it extensively, for every thing they can think of. In most cases, it will not work, or it will not work well, but the point is to try and learn.

I especially do this with my young kids, as I imagine that they will go into a workforce that will probably use AI a lot. I certainly don’t know what careers will exist in the future and how they are going to be affected by AI, but I want them to be very familiar with the usefulness and limitations of AI so that they are well-equipped to make the best decisions for themselves.

As a society, it’s beyond time to start thinking about the allocations of the gains and losses. One thing is clear: if nothing changes, the losses will be allocated to workers that have their job automated. Even at the time of the Luddites, there were already proposals to allocate resources to retrain workers that were losing their jobs to machines. These proposals ended up not being implemented.

But since the time of the Luddites, we have accumulated a lot of knowledge, and we have also improved our communication infrastructure. We can share stories like this one with thousands of people. People can find training online. People have access to cheap AI, even free. We are much better equipped this time around.

Hopefully, this time, we can do better.

References

The main source for this article was the excellent Cautionary Tales podcast episode by Tim Harford, General Ludd’s Rage Against the Machines. I’ve also used the following sources:

• The Rising of the Luddites
• Daron Acemoglu interview in “People I Mostly Admire
• Power and Progress
• The Luddite Link

4-week checkpoint

The doctor and I agreed to run the treatment for about four weeks, after which I should check back for any adjustments. I agreed to document how things were going to make the adjustments easier.

For the first four weeks, I diligently took the prescribed Ritalin dose, usually at 8 AM. As you would expect with someone with ADHD, I forgot to take the medicine a few times. When that happened, I usually remembered before 11 AM. During the 28 days, I completely forgot one time, took it by 11 AM five times, and took it at 1 PM one time.

I noticed a few things when taking the medicine. The first is that I flush a little. When I looked at online forums like Reddit, I found that this seems to be a common side effect that disappears over time. The second is that I usually feel wiped out at the end of the day, even more than usual. Interestingly, this helped me sleep better.

The other side-effect we talked about was insomnia. Taking the medicine at 11 AM didn’t seem to make any difference in my tiredness or productivity, but on the day I took the medicine at 1 PM, I was awake until 1 AM, which is unusual for me (I’m usually up early at around 6 AM). Insomnia is a known side-effect of stimulants, and definitely something to monitor. After that experience, I decided to completely skip the medicine for days I forget to take it before 11 AM.

Because I keep track of almost every one of my tasks in a to-do list, I was able to notice that there was some small improvement of my symptoms, but it was minimal. The main improvement was that I was able to tackle some tasks that were “annoying but easy”, like scheduling someone to clean the dryer vents of my house. Incidentally, Thumbtack (no affiliation) has been pretty good for scheduling those “annoying but easy” home maintenance tasks. As for the other symptoms, my overall levels of forgetfulness and fidgetry didn’t seem to change.

I met with my doctor again, and she suggested increasing the dose to 20 mg of long-acting. which is the default dose for most people. I will start with this new dose on Monday. The long acting version is supposed to work for 8-12 hours.

Therapy

The therapist I’m working with can only meet every other week. I think that now, in the beginning, it would be beneficial for me to meet more frequently, especially as try to recast a lot of the things that have been problematic for me over my life under the new lens provided by a diagnosis. For example, sometimes I would think that I was not interested in some things, but it was just a period of my life where a lot was going on and I was easily distracted. We are still trying to find out a schedule that works well for both of us.

Verdict

I think the initial doses of therapy and medicine were not sufficient for improving the symptoms substantially, but at least the side-effects haven’t been bad. Now that the medicine dose has increased, I plan to increase the frequency of my therapy and document my progress after three months.

Multiple open projects

As counterintuitive as it may seem, I usually have multiple projects going on at the same time. The way this works is that when I’m working on a project and get distracted, the distraction usually comes from another project (like an email or a meeting), so I switch to that project. This way, I’m still being productive.

Log activities as I start them

I got the habit (or routine) of logging most activities as I start executing them. This helps me when I eventually get distracted, because I can go back to the log and see what I was doing before I got distracted. I use Siri and Microsoft To-Do to log activities, as described in the Coping Technologies section.

Plan activities I don’t want to start

I learned this technique recently. When I realize that I haven’t been productive and my day is a “bad day”, instead of trying to concentrate and do more, I go into planning mode. I essentially give up trying to do things, I start just planning and preparing things. If I’m trying to download 100 images and keep getting distracted, I would instead just list the websites that I want to download the images from, create the the Azure storage to receive it, etc. I may call this productive avoidance.

Microsoft To-Do

I use Microsoft To-Do to keep track of tasks. I like it because it’s very simple, it’s multiplatform, so I can see and check tasks from my phone, my computer and my watch. Since it integrates well with Siri, I can add tasks simply by saying ‘Hey Siri, remind me to X’, which helps me keep track of things. I specifically add tasks to my to-do list as I’m starting them, because that will help me remember to finish if I get distracted.

Apple AirTags

I have Apple AirTags on everything since I lose something almost daily. The new AirTags have a feature called Precision Finding that allows me to find things a lot faster, and I use it a lot.

Wallet with Key Ring

I bought a wallet with an attached keyring that allows me to carry my badge, car and house keys, wallet, and an AirTag all together, making it hard for me to lose things.

S-biner carabiners

I have a lot of S-biner carabiners. I use them to attach things to my backpack and to each other, so I have only one bundle of items to keep track of.

Ray-Ban Meta Sunglasses

The Ray-Ban Meta works as sunglasses, a headset and as reading glasses. It also has a camera, which I use quite a lot, and an AI that I don’t use as much yet.

Kindle Scribe

I always carry a Kindle Scribe. I had (and lost) many notebooks and loose sheets of paper. The Kindle Scribe is an e-ink notebook that feels like a real notebook, but has a lot of capacity. I can also carry academic papers on it. I used to have a reMarkable 2 but I changed to the Kindle Scribe because it solves a few more of my use cases.

Pillbox

I use a pillbox to organize my vitamins and medications. I usually forget whether I took the medication or not a few hours after the time I was supposed to take it, so I use the pillbox to keep track of whether I took it or not.

Coping Strategies

I also have a few coping strategies that I use to manage my symptoms. I will list them below:

Prerequisites

To run the code in this article, you need to have Python 3.11 installed. This will enable you to use the TaskGroup functionality. You also need to have the openai Python package installed, version 1.0 or higher.

In my case, I have two Azure OpenAI subscriptions, one in the US East region and another in the US West region. Therefore, I have two API keys, one for each region. To make things more interesting, I’ll assume that each region has different models, gpt-4 in US East and gpt-4-turbo in US West.

To test concurrency, I will ask GPT to perform a task that takes about 30 seconds to complete, such as generating a five-paragraph story about a theme. I have four themes, “dog”, “cat”, “chicken”, and “tiger”. I will ask GPT to generate stories about “dog” and “cat” concurrently, and then “chicken” and “tiger” will be generated concurrently. I will time how long it took to generate the stories concurrently, and I will also measure the time to generate the stories sequentially, so that I can compare the results.

The code

First, let’s import the required libraries.

import os
from openai import AsyncAzureOpenAI
from dotenv import load_dotenv
import asyncio
from timeit import default_timer as timer

Note that I’m importing the AsyncAzureOpenAI class from the openai package. This class is used to call the OpenAI API asynchronously. I’m also importing the load_dotenv function from the dotenv package, which is used to load environment variables from a .env file, where my subscription keys and endpoints are stored.

The concurrency is managed by the asyncio library. I’m also importing the timer function from the timeit package to measure the time it takes to generate the stories.

Now, let’s define the function that will call the OpenAI API to generate the stories.

async def call_openai(client, id, model, theme, answers):
    print (f"Generating a story about {theme} using {id}.")
    response = await client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": f"Generate a five-paragraph story about a {theme}."},
        ]
    )
    answers.append(response.choices[0].message.content)
    print (f"Generated a story about {theme} using {id}.")

The function is a straightforward call to the chat.completions.create method of the AsyncAzureOpenAI class. Note that I’m requesting a five-paragraph story about a theme. I chose that prompt because it takes about 30 seconds to complete, so even if GPT randomly generates a shorter story, we should still be able to see the differences. The response is appended to the answers list (not shown).

Let’s now see the main body of the function.

async def main():
    load_dotenv()
    client1 = AsyncAzureOpenAI(
        api_key=os.getenv("USEAST_KEY"),  
        api_version="2023-12-01-preview",
        azure_endpoint = os.getenv("USEAST_ENDPOINT")
    )

    client2 = AsyncAzureOpenAI(
        api_key=os.getenv("USWEST_KEY"),  
        api_version="2023-12-01-preview",
        azure_endpoint = os.getenv("USWEST_ENDPOINT")
    )

    themes = ["dog", "cat", "chicken", "tiger"]
    i = 0
    model1 = "gpt-4"
    model2 = "gpt-4-turbo"
    id1 = "client-1"
    id2 = "client-2"

In the code above, I creaate two separate OpenAI clients, one for each region. I also define the themes for which I want to generate stories, and the models that I want to use. I also define the IDs for each client, so that I can identify which client generated each story.

Calling the functions sequentially

Now let’s call the functions sequentially, so that I can establish the baseline.

The code below calls the call_openai function (that we defined above) sequentially, and measures the time it takes to generate the stories. The code is still inside the main function, which is called at the end of the script.

    answers_s = []
    start = timer()
    # Run sequentially
    for i in range(0, len(themes), 2):
        start_step = timer()
        await call_openai(client1, id1, model1, themes[i], answers_s)
        await call_openai(client2, id2, model2, themes[i+1], answers_s)
        end_step = timer()
        print(f"Finished generating stories about a {themes[i]} and a {themes[i+1]} sequentially in {end_step - start_step:.2f} seconds.")
    end = timer()
    print(f"Generated stories sequentially in {end - start:.2f} seconds.")

Generating a story about dog using client-1.
Generated a story about dog using client-1.
Generating a story about cat using client-2.
Generated a story about cat using client-2.
Finished generating stories about a dog and a cat sequentially in 85.34 seconds.
Generating a story about chicken using client-1.
Generated a story about chicken using client-1.
Generating a story about tiger using client-2.
Generated a story about tiger using client-2.
Finished generating stories about a chicken and a tiger sequentially in 57.66 seconds.
Generated stories sequentially in 143.00 seconds.

Calling the functions concurrently

Here, I will use the asyncio.TaskGroup class to run the call_openai function concurrently. This is still inside the main function.

    # Run concurrently
    answers_c = []
    start = timer()
    for i in range(0, len(themes), 2):
        start_step = timer()
        async with asyncio.TaskGroup() as tg:
            print(f"Started generating stories about {themes[i]} and {themes[i+1]} concurrently.")
            tg.create_task(call_openai(client1, id1, model1, themes[i], answers_c))
            tg.create_task(call_openai(client2, id2, model2, themes[i+1], answers_c))
        end_step = timer()
        print(f"Finished generating stories about a {themes[i]} and a {themes[i+1]} concurrently in {end_step - start_step:.2f} seconds.")
    end = timer()
    print(f"Generated stories concurrently in {end - start:.2f} seconds.\n\n")

The TaskGroup class is a context manager that allows you to run multiple tasks concurrently. You can start asynchronous tasks with the create_task method, and the context manager will wait for all tasks to complete before exiting. In my case, since I have two clients, I will start two tasks in each iteration of the loop.

Let’s see the results.

Generating a story about dog using client-1.
Generating a story about cat using client-2.
Generated a story about cat using client-2.
Generated a story about dog using client-1.
Finished generating stories about a dog and a cat concurrently in 34.89 seconds.
Started generating stories about chicken and tiger concurrently.
Generating a story about chicken using client-1.
Generating a story about tiger using client-2.
Generated a story about chicken using client-1.
Generated a story about tiger using client-2.
Finished generating stories about a chicken and a tiger concurrently in 30.25 seconds.
Generated stories concurrently in 65.14 seconds.

if __name__ == "__main__":
    asyncio.run(main())

What if you have just one subscription?

The code above also works if you only have one subscription. For example, if you were to replace the client1 with client2 everywhere in the code above, you would still be able to run the tasks concurrently. The only difference is that you would be using the same subscription for both tasks, so you would be consuming your rate limits faster. However, you would still be able to run the tasks concurrently, and you would still see a significant improvement in the time it takes to generate the results.

Conclusion

Running AI models concurrently can significantly reduce the time it takes to generate the results. This is especially important when you have to generate a large number of results, or when the results take a long time to generate. In this case, we saw that running the models concurrently with double the resources, as expected, took about half the time. This is a significant improvement, and it can make a big difference in practice.

The first week (Jan 12 - Jan 19)

Switching from a diagnosis to a treatment was harder than I expected. After I finished the diagnosis, my results were immediately sent the results to my primary care provider (PCP). A few days went by and nothing happened. I emailed my PCP’s care coordinator, and she asked me to send a copy of the diagnosis to her. I did, and again nothing else happened for a few days. Then I found an opening in my primary care provider’s calendar and booked it for next day. My PCP had not yet received the results from her own coordinator, so I printed a copy and brought to her.

How are things going so far?

Getting diagnosed is helpful, because now I understand many of the things that happen, even though I still have trouble controlling them. As of now, the dosage of the medicine is probably too low to make a difference, but I noticed that I am getting more tired than usual at night and sleeping better. My level of distraction and being overwhelmed don’t seem to have improved substantially.

The post above is about my first week of treatment. I also documented the first four weeks of treatment.

PhD fields are vastly different

You may have talked to someone or heard their experience with a PhD, and you may have heard that it was a great experience, or that it was a terrible experience. Before you incorporate that information into your decision, you need to know what the person’s field was.

Although there are some similarities between PhD fields, in practice the careers in different fields are vastly different. The differences are not only between fields, but also between countries, universities, departments, and even specific topics. Someone’s experience with a PhD in English is going to be vastly different from someone’s experience with a PhD in Finance or Accounting. Even within the same field there are differences. The experience of someone in Finance — Investments is going to be different from the experience from someone in Finance — Real Estate. The supply and demand for professionals in different fields varies a lot. Therefore, when you are looking for opinions, it’s important to talk to people that at least have done a PhD in a similar subject as you are considering, and as close as possible to what you are actually considering. At a high level, different fields can be thought as completely different careers. Asking an English PhD about a PhD in Finance is like asking a poet about a career in accounting.

Some fields have a robust market in the industry, but for some others, the market is mostly limited to academia. For example, PhDs in Finance, Accounting, Economics, Statistics, Computer Science, and Engineering have a robust market in the industry. On the other hand, the industry market for PhDs in English, History, and Philosophy is very limited. This is not to say that there are no jobs, or that the PhD won’t be helpful to get a specific job, but that the PhD is usually not a requirement for industry jobs in these subjects and it’s unlikely to help in the average case. In addition, some careers are more careful about admitting PhD students, and only admit a small number that will be employable later (that tends to be the case with Finance and Accounting). Some other fields admit as many students as possible, which can cause the market to become oversupplied.

What is your objective?

If you worked with me on anything, you know that this is a question you hear a lot from me. You need to know what you’re trying to accomplish before you can be helped.

If you want to work as a researcher in a research university, you need a PhD, and usually a PhD from a well-regarded university. For most other cases, a PhD is not a hard requirement. There are some positions in large companies that require it, usually research positions, and there are some positions for which it is strongly desired. In some cases, requirements make little sense: I’ve seen someone with a Ph.D. in Philosophy from a mid-tier university get an interview for a software development position that required a PhD, but another person with a Master’s in Computer Science from a top university not get an interview for the same position because they didn’t have a PhD, even though the person without the PhD had a lot of experience in the subject and in the area. These cases are uncommon. One example of an industry position that often requires PhDs is the Applied Scientist position at Amazon, although if I recall correctly, it’s not a hard requirement for all teams.

If you don’t want to work in a position that requires a PhD, your question is probably about the overall value of the PhD, as in “is it worth the investment?” Most PhDs are funded with a stipend that is enough to live on, but it’s not a lot of money. This makes getting a PhD a substantial investment — depending on your career, you will be forfeiting 4-6 years of salary, and you will be spending a lot of time and effort. The starting salary of a Computer Science graduate in the West Coast is easily over $100,000, so you are looking at a $400,000 to $600,000 investment in forfeited salary alone, and there’s no guarantee that having a Ph.D. will substantially improve your salary later. For science careers with a solid industry market, the PhD is more likely to make a difference in your salary after the initial investment than in other fields. For a long career in these fields, it’s likely to pay off, but not guaranteed. For fields with a weaker market, a PhD is unlikely to pay off financially. Also, doing a PhD outside of your career is unlikely to help you get a job. For example, if you are planning to work as a software engineer and want to do a PhD in English before you start your career, you are probably going to be better off financially if you don’t do the PhD. It may still be worth it for you for the learning experience.

If want to do it because you really love to learn… well, maybe that’s a good reason. PhDs in top universities are very rigorous, and you will learn a lot. I have to say that after doing it, you will feel that you know less than before you started it, but at least you are going to be pretty sure about what you know well and what you don’t. You get lots of skills: you will learn to research rigorously, you will improve your presentation skills, you will learn how to learn, you will get project management skills, but those are skills that you can get elsewhere while being paid.

Can it help me get a job?

As it’s common with most answers… it depends. Let’s do a multiverse analysis. Assume that there are two versions of you. Both just finished a Computer Science degree at a prestigious university. One of them will go to work in a well-recognized tech company for five years, the other version will go for a PhD in Computer Science at a prestigious university for five years. Five years later, both apply for a non-research software engineering position. Besides having more money, the engineering version of you has more experience and it’s more likely to get the job. On the other hand, if five years later both of you apply for a research position in a tech company, the PhD version of you is more likely to get the job. Only the research version of you can apply for a tenure-track Assistant Professor position. Ten years after finishing school, the software engineer version is likely to have more wealth.

What about the experience?

For most people in research institutions, the experience is very hard. It’s a lot of work, and it’s very stressful. It’s not uncommon to have to work 60-80 hours a week. There’s a lot to learn and a lot of work to do in a short amount of time, and your work will be evaluated at the highest level. You need to improve the overall human body of knowledge, and a lot of dedicated people have been working on it for a while before you started, so it’s supposed to be hard. You need to find something that you enjoy doing hours on end and researching very deeply. You need to be able to self-motivate. For some careers, a lot of the experience is about recovering from failure, as a successful researcher has many papers rejected. Most people who start a PhD are good students that are used to success, and it’s a very different experience to have to deal with failure.

Therefore, it’s not something that most people would recommend for the experience alone.

For people in non-research institutions that get the title without having to publish new research at top tier journals, the experience tends to be a lot better, but the applicability of the degree also tends to be a lot lower.

So, should I do a PhD?

If you want to be a researcher in a top university, it’s kind of the only way. If you are thinking about doing a PhD to make someone else happy, or to get a lot of easy money, my suggestion is that you shouldn’t do it — it’s an uncertain way to make parents happy, or to make a lot of money. For all other cases, the answer is an unsatisfactory “it depends a lot on your particular situation and unlikely that you’ll find the answer in an online article”. You need to look hard into your objectives and talk to some people in both your subject area and the career you plan to pursue afterwards. A PhD is probably one of the major purchases you will do in your whole life, usually as big as a house, so you should invest a lot of time in researching it and making sure it’s the right decision for you. It’s definitely worth your time and money to get information from people that have been through a similar path in order to answer your questions.

What problems I want to solve?

These days, I can take handwritten notes in most of my devices. My main device is a Surface Laptop Studio 2, which folds into a tablet and has a stylus. I also have an iPad Pro 13” with a stylus, but my Surface mostly replaced it. However, I cannot do everything with my Surface Laptop Studio 2. Here are a few of the issues preventing it from being my only device:

• It is a little on the heavy side. It is heavier than the iPad Pro 13”, that many people already find too heavy for handwritten notes.
• Battery life is not great. I can get about 2-3 hours of battery life out of it, which is not enough for a full day of meetings.
• It is easy to get distracted by notifications and other apps on the device.

How does the Kindle Scribe solve these problems?

The Kindle Scribe solves these problems in the following ways:

• The Kindle Scribe is light. It is 433 grams (28g heavier than the reMarkable 2) and 5.8 mm (0.22inches) thick, 1.1 mm (0.02 inches) thicker than the reMarkable 2. For comparison, my iPad Pro 13” is 682 grams (1.5 pounds) and 5.9 mm (0.23 inches) thick. I can confortable use the Scribe to read in in bed, which I do often.
• It has a very long battery life. I charge it about once a week.
• The Kindle Scribe doesn’t bother me with notifications, and its browser is hidden and very limited, so I don’t get distracted by it. One advantage of having a browser is that you can use captive networks, such as the ones in hotels, which was a problem I had with the reMarkable 2. Frequently I had to connect to my phone’s hotspot to get my notes out of the reMarkable 2. I have not traveled with the Kindle Scribe yet, but I expect it to be easier to join captive networks.

What is the Kindle Scribe useful for?

I have been using the Kindle Scribe for the following use cases:

• Academic paper reading and annotation: This was my main goal, and where the Kindle Scribe succeeded and the reMarkable 2 fell a little short because of the lack of contrast and backlight. There is still one issue with the Kindle scribe: similar to the reMarkable, it has a 10.3” screen, smaller than A4 paper, so I need to read papers in landscape mode and scroll. I can write notes directly on the PDF, and highlight parts of the PDF. The highlights are saved to a file called “My Clippings”, in text format.
• Reading books: A big advantage that the Kindle Scribe has over the reMarkable 2 is that the Scribe has my whole Kindle library. Although some books I read require the iPad, the vast majority are perfectly fine in the Scribe, and this has helped me read more.
• Journaling: I have been using the Kindle Scribe for journaling almost every day. The writing experience is good. If I never had a reMarkable, I think I’d find it great, but the reMarkable definitely feels more like paper. The Kindle Scribe sometimes feels like it slides a little bit too easily, but it’s a lot better than my experience writing on the iPad or on the Surface Laptop Studio. One thing that helped was to change my default pen to “marker” or “fountain pen”.
• Note-taking in meetings: I have taken the Kindle Scribe to meetings, and this allows me to focus a lot better. I have had no problem importing notes later - you can scan them and send them by email to yourself if you need to use your notes in an email, for example.
• Writing down ideas: I have been using the Kindle Scribe to write ideas, similar to what I did with the reMarkable. I find it easier to use the Kindle Scribe, because I can do it in low light conditions (e.g., when my kids are watching a movie in the living room).

What are its downsides?

Here are some of the downsides:

• Templates: the Kindle Scribe doesn’t have many templates. My preferred template on the reMarkable 2 was the “small dotted”, a template that had barely visible dots to guide lines The Scribe has a dotted template but the dots are quite big, I end up mostly using the blank template. Also, the reMarkable 2 has tons of interesting templates (e.g. “Cornell Notes”) for download. I couldn’t find those for the Kindle yet.
• Size: Its 10” screen is a little small for reading PDFs. I have been reading a lot of academic papers on it, and it’s easier to do it on landscape mode, where it gets a little bigger, but requires scrolling. This is similar to my experience with the reMarkable 2.
• Writing experience: This is very subjective, but the writing experience is not as good as the reMarkable 2. The reMarkable 2 feels more like paper, and the Kindle Scribe feels a little more slippery. However, it’s still a lot better than writing on the iPad or on the Surface Laptop Studio.
• Drawing mode: In the reMarkable, if you hold the pen for a few seconds, it switches to drawing mode, where you can draw shapes. I used this a lot to draw boxes and arrows. The Kindle Scribe doesn’t have this feature, so I have to draw boxes and arrows by hand with my low drawing skills.
• No page turn buttons: some Kindles have page turn buttons, but the Kindle Scribe doesn’t. I don’t mind this, but some people do.

Conclusion: do I recommend it?

I feel much better recommending the Kindle Scribe than the reMarkable 2 to most people. Besides solving more problems, the Scribe is cheaper than the reMarkable 2. I bought my Kindle Scribe with a premium pen for $299, while the reMarkable can easily get to $429. The materials used in the Kindle Scribe feel cheaper than the reMarkable 2, but it has more storage, a backlight, and access to the Kindle store. Being pragmatic, I think the Kindle Scribe is a better device for most people.

If your regular Kindle is due for an update, it may be worth it to get the Kindle Scribe instead. I think it is a good device for people who want to read academic papers and books, and take notes on them. It is also a good device for people who want to replace their Moleskines and loose paper sheets.

You can buy the Kindle Scribe at Amazon.

Symptoms

Whether I have ADHD or not, I do have some symptoms that are commonly associated with ADHD:

• Difficulty getting started with tasks, even simple ones. I have gone to appointments I didn’t need to because I didn’t want to call to cancel the appointment.
• Difficulty putting the finishing touches on tasks. It takes me 90% of the time to do 90% of the work, and 90% of the time to do the remaining 10% of the work.
• I lose an incredible amount of things. When I was a kid, my mom was once told that about 10% of the items of the lost and found in a 300-student school were mine.
• I have a hard time focusing on things that are not interesting to me.
• I oscillate between hyperfocus and not being able to focus at all.

Despite the difficulties, I manage to function well, because I developed a lot of coping mechanisms and technologies over the years.

Why I Decided to Get Tested

The flurry of posts in Threads due to the ADHD awareness month in October 2023 got me started. Several posts described experiences that were very similar to mine. A particular story that resonated a lot with me was the story of someone who was treated for anxiety for years when they in fact had ADHD, and only got diagnosed in their 40s.

I found out that this is a common problem. I have been undergoing treatment for anxiety, and though there was some improvement, progress has been slow. Several posts described a situation similar to mine: people that made little progress while being treated for anxiety, but once diagnosed with and treated for ADHD as adults, their situation improved significantly.

This gave me hope, prompting me to research the diagnostic process. Starting the process ended up being more confusing than helpful. The information I found online described several different processes and tests, some of which even seemed to promise a diagnostic outcome. The best advice came from Threads and it was clear: begin by consulting a medical professional and expressing your concerns, and go from there.

The Referral

The process, at least where I live, seems to be designed to make it hard for people with ADHD to get diagnosed. I had to go through several steps, and I’m still not done. I live in the state of Washington in the United States. The experience may be different in other states or countries.

As mentioned above, the first step is to talk to a medical professional. I had a previously scheduled appointment for my annual physical just a few weeks after I decided to get tested, so I waited for that appointment in mid-October. Trying to schedule a separate appointment would have taken a lot longer.

In the appointment, my PCP told me they would call me with a referral in a couple of days, but instead of calling me, they called my emergency contact, who forgot to tell me about it. Luckily, people in Threads kept reminding me to follow up, and I called my PCP office a few days later. I found the referral and called the psychologist office.

The Initial Evaluation

I called the psychologist’s office in early November. The first thing I needed to do was to set up an account in their system, which took a few days. Then, when the account was completed, they asked me to fill several standard medical forms, plus a two-page biography describing why I think I may have ADHD, and three questionnaires designed to assess whether I have ADHD and potentially autism, which tends to happen hand-in-hand. The questionnaires are below:

1. Adult ADHD Self-Report Scale (ASRS)
2. Brown ADD Scales
3. Autism Spectrum Quotient (AQ)

The first questionnaire is a screening tool for ADHD. The second is a more detailed questionnaire about ADHD symptoms. The third is a screening tool for Autism Spectrum Disorder (ASD), a common comorbidity. The first six questions serve as a screening tool: if you score in the grey areas for four out of the six questions, you should definitely talk to a medical professional.

I completed the forms December 8th. After all the forms are filled, there are three more appointments to go through, and the first can only be scheduled after the forms are completed. I received a message on the day after that I should schedule the first appointment. I only managed to call back on December 22nd, and scheduled the first evaluation for January 3rd, 2024.

The First Meeting

On Jan 3rd, 2024, I had the first meeting with the psychologist. We introduced ourselves and he asked me about the symptoms that resulted in me seeking a diagnosis. I described the symptoms I listed above, and he asked me to elaborate on each one of them. He also asked me about my childhood, and I described how I was a very good student, but I had a hard time focusing on things that were not interesting to me. I also described how I lost a lot of things, and how I had a hard time getting started with tasks.

He asked about my family history. My mom is definitely not neurotypical, but she has never sought a diagnosis. I did not have enough contact with my dad to know if he was neurotypical or not. He also asked about my children, and I told him that their school has guided me to request accomodation arrangements for them (these are called 504s). I told him that I plan to get them through an ADHD and autism evaluation as well and that the school counselor had already suggested that earlier in the year.

The whole meeting lasted about 40 minutes. We scheduled the appointment for the second meeting on January 8th, 2024.

He then gave me three more tests to fill out on my own time, which I did later at night on that same day.

• A personality test with 344 questions: the test asked a lot about whether I can control other people’s minds, whether I think they control my mind, and how frequently I think about suicide. It also repeated several questions about the usual ADHD symptoms. It would ask “Do you always feel distracted?” and offer four options for me to choose (False, Slightly True, Mostly True, True). Later, it would ask “Do you frequently feel distracted?”, and show the four options again. This one took a while to get through.
• The Big Five personality test: a standard personality test with about 50 questions, which was somewhat quick to get through.
• An attention test called IVA-AE2: this test repeatedly shows an image of a number and speaks another number. For example, it can show 2 and speak “six”. You are assigned two different target numbers, one to click when you see it and another to click when you hear it. It then speaks and shows random numbers for 20 minutes. For example, if you are assigned the number “3 when seen” and “5 when heard”, you should click if it shows 3 and speaks 6, you should also click if it shows 2 and speaks 5, but you should not click when it shows 5 and speaks 3. I think it’s pretty normal to make the mistake of clicking when you hear the number you’re supposed to click only when you see and vice-versa, and I’m sure I made plenty of mistakes, but I think most people would. This test was very tiring.

The Second Meeting

On January 8th, 2024, I had the second meeting with the psychologist. This meeting was markedly shorter. He sent me an email with a link to the MOXO d-CPT ADHD Assessment, a test that has visual and auditory distractions, for me to do during the meeting, while he scored my previous tests.

The test is similar to the IVA-AE2 test I did for the first meeting. It repeatedly shows images of several playing cards, but I have to identify a specific one (the Ace of Hearts) and press the space bar when I see it, while a lot of distractions happen simultaneously. The test takes 18 minutes to complete, and is also somewhat stressful and tiring.

I downloaded a demo from their website and edited the video to show a 15-second snippet of the most difficult part below.

After I completed the test, he asked me a few more questions:

• Do I play videogames a lot? (I used to, but I don’t anymore)
• How did I feel about the IVA-AE2 test? (I felt it was very tiring)
• Did I have any childhood trauma? (I wouldn’t call it trauma but my father left my life quite soon, and my mom is not neurotypical, and we have some money difficulties, which led me to work really early in my life)

I imagine the first question was related to my reaction time, but I’m probably only going to learn about that in the third meeting.

We ended the second meeting somewhat early, after just 30 minutes, and scheduled the feedback meeting for Wednesday January 10th, 2024.

The Third Meeting: The Diagnosis

The doctor I used, Dr. Richard Wemhoff, Ph.D., from Emmaus Counseling Center, is a specialist in adult ADHD diagnosis. If you read the whole process above, you will see that it was very comprehensive. He goes through ten steps, seven of which are questionnaires. I’ll describe the process and the final diagnosis below.

1. Do your symptoms get in the way of your personal or work life? Although this is a simple question, it’s an important one. For me, the answer was yes.
2. Did this happen when you were young? Again, this was true in my case. ADHD that appears in adulthood is unusual, and it’s important to rule out other causes.
3. Brown ADD Scales: this test goes from a scale of 0 to 120. 0-40 means ADHD is unlikely. 41 to 55 means is possible, and 56+ means it’s likely. I scored 67, and it was significant in all the five components: attention, activation, emotion, memory, and effort. One interesting thing is that if some components are too significant, this may indicate a different condition. For example, a high score on emotion (meaning that the person is feeling very bad about the symptoms) may indicate depression. That was not my case in any of the components.
4. DSM-5: a set of 18 questions, 9 to screen for inattention and 9 to screen for hyperactivity. I scored 7/9 on inattention and 8/9 on hyperactivity, again, indicative of ADHD. There are three types of ADHD: innatentive, hyperactive, and combined. I scored high on both, so, according to this test, I’m likely to have combined ADHD.
5. Can the symptoms be due to another condition? This is where the long tests done after the first meeting come into play. He scored the results and I don’t seem to have any other condition. The only thing that was a little above average was “social detachment”, but still within normal range.
6. Family history: ADHD tends to run in families. My mom has some condition, as of yet undiagnosed. My kids have ADHD symptoms and given a screening, they should be tested, so there’s definitely a family history.
7. Adult ADHD Self-Report Scale (ASRS): an 18-question test, similar but not exactly the same as DSM-5. This also resulted in a high score, and it’s a good indicator of ADHD.
8. MOXO d-CPT ADHD Assessment: this is the test I did in the second meeting, and is considered to be very precise. I scored “ADHD present”. The test has four components. I only had a normal score on “impulsiveness”, meaning I’m not overly impulsive. I scored below average on “attentiveness”, and far below average on “timeliness” and “hyper reactivity”. This last one means that I overreact to distractions, and I felt it in my soul.
9. IVA-AE2: to everyone’s surprise, I passed on this attention test. The doctor said that this is not that unusual, people with ADHD can hyperfocus.
10. Autism Spectrum Quotient (AQ): this is a test to screen for autism, not to diagnose it. Autism is present in 25% of the ADHD cases, so it’s important to rule it out. I scored way above the cut-off range, which means that I also need to be evaluated for autism.

In summary, the result is that I definitely have ADHD, and that information was sent to my PCP to discuss treatment options, which may include medicine. I also need to be evaluated for autism, which is a different process that I will document in a different article. The doctor also said that I should be proud of myself for having accomplished the things I did in life given my condition, and that things should improve a lot when I start treatment.

I am describing my process with the treatment at this article.

Do I recommend this process?

During this whole process, I felt conflicted. End-to-end, it took months, and for a while I was so distracted with other things that I ended up delaying my diagnosis. I heard stories of people that get their diagnosis very quickly, in a single appointment. My process had several steps, and I had to do a lot of tests, and I was not sure if it was going to be worth it.

There is, however, an immense advantage in going through a process that was so comprehensive. I feel very confident about the diagnosis. I know that I have ADHD, and I know that I don’t have any other condition that masks the symptoms. And now I know what to do about it.

Reactions

I posted about my experience in Threads, and I got a lot of reactions. Most people are supportive and describe their own experiences being diagnosed. Some people, however, are skeptical. They say that ADHD is overdiagnosed, and that it’s a fad, or a quick excuse to be lazy and forgetful. While I disagree, I have seen some websites that promise a “quick and easy” diagnosis and to get you medications fast. It feels similar to advertisements for Viagra and off-label Ozempic.

Overall, having documented this process through posts has been a good experience, and I really hope it helps other people.

Treatment

I have been documenting the progress with my treatment. I have written a 1-week update and a 4-week update.

What about you?

If you suspect you may have ADHD, I recommend that you talk to a medical professional. Your PCP is a good place to start. Online communities, such as the ones in Threads, can also be a good place to start.

Configuring Microsoft Entra ID and Linking the App

Ultimately, I want to ensure that users are only allowed to see the app if they are authorized. To do this, I will use Microsoft Entra ID to authenticate users. To get to that, we first need to authenticate the users, and then authorize them to access the app. This post is about the authentication part.

To create an app that does authentication, I followed the instructions at Quickstart: Sign in users and call the Microsoft Graph API from a Python web app. The instructions will tell you where to find the CLIENT_ID and how to generate a CLIENT_SECRET for the app. The AUTHORITY needs to be set to AUTHORITY="https://login.microsoftonline.com/<tenant_id>", which I obtained from the Azure Portal, in the Overview page of Microsoft Entra ID.

I downloaded the full code for the app, and the only change I had to make was to create an .env file with the variables described above. This is not required for the app to run on the web, but it helps when running locally.

The .env file should look like this:

CLIENT_ID=[client_id]
CLIENT_SECRET=[client_secret]
AUTHORITY=https://login.microsoftonline.com/[tenant_id]

The instructions don’t mention this, but after finishing the instructions, you also have to link the Entra ID configuration to the Web App you created in the first step. To do this, go to the Azure Portal, open the App Service, go to the “Authentication” tab, and click on “Microsoft” under “Identity providers”. Then, select the “Existing App” tab and select the app you created in the previous step.

The final connection step was to set up the environment variables CLIENT_ID, CLIENT_SECRET and AUTHORITY in the App Service. I followed the instructions on this link to set the environment variables in the Web App to the same values present in the .env file.

Once I did all this, I pushed my code to GitHub and it was automatically deployed to the App Service.

Code walkthrough

The code is for a Flask Web App. Flask is a Python framework for building web apps. I decided to use it instead of other alternatives like Angular or NextJS because it allows me to use Python, which is the language I’m most familiar with. The code is in the app.py file, and I will walk through it below. I also created an app_config.py file to store the configuration variables, which you can see in GitHub here. I will not walk through the app_config.py file, because it is just a file to store the configuration variables.

import identity.web
import requests
from flask import Flask, redirect, render_template, request, session, url_for
from flask_session import Session

import app_config

__version__ = "0.7.0"  # The version of this sample, for troubleshooting purpose

app = Flask(__name__)
app.config.from_object(app_config)
assert app.config["REDIRECT_PATH"] != "/", "REDIRECT_PATH must not be /"

Session(app)

# This section is needed for url_for("foo", _external=True) to automatically
# generate http scheme when this sample is running on localhost,
# and to generate https scheme when it is deployed behind reversed proxy.
# See also https://flask.palletsprojects.com/en/2.2.x/deploying/proxy_fix/
from werkzeug.middleware.proxy_fix import ProxyFix
app.wsgi_app = ProxyFix(app.wsgi_app, x_proto=1, x_host=1)

app.jinja_env.globals.update(Auth=identity.web.Auth)  # Useful in template for B2C
auth = identity.web.Auth(
    session=session,
    authority=app.config["AUTHORITY"],
    client_id=app.config["CLIENT_ID"],
    client_credential=app.config["CLIENT_SECRET"],
)

@app.route("/login")
def login():
    return render_template("login.html", version=__version__, **auth.log_in(
        scopes=app_config.SCOPE, # Have user consent to scopes during log-in
        redirect_uri=url_for("auth_response", _external=True), 
        prompt="select_account",  
        ))

@app.route(app_config.REDIRECT_PATH)
def auth_response():
    result = auth.complete_log_in(request.args)
    if "error" in result:
        return render_template("auth_error.html", result=result)
    return redirect(url_for("index"))

@app.route("/")
def index():
    if not auth.get_user():
        return redirect(url_for("login"))
    return render_template('index.html', user=auth.get_user(), version=__version__)

if __name__ == "__main__":
    app.run()

Testing

I navigated to the app URL and was able to sign in with my Microsoft account. I was then redirected to the app, where I could see the information about my login.

Now that I have an app that can authenticate users, I need to work on authorization, so that only authorized users can access the app.

The code

My blog is written in Quarto, which is a Markdown-based document format. I use Quarto Markdown (.qmd) files to write my blog posts. A .qmd file is a Markdown file with YAML front matter. The YAML front matter contains metadata about the document, such as the title, author, and date. I wanted to add the description meta tag to the YAML front matter of each file.

The first thing to do is to extract the YAML front matter from the file. I used a regular expression to do this. If the file doesn’t contain YAML front matter, I will skip it.

import yaml
import re
import os
from openai import OpenAI
from dotenv import load_dotenv

def parse_qmd(content):
    # Extract the YAML front matter
    matches = re.findall(r'^---\n(.*?)\n---', content, re.DOTALL)
    if not matches:
        return None, None
    return matches[0], content

I wrote a function to recompose the file content after modifying the YAML front matter. This function takes the original YAML front matter, the modified YAML data, and the full file content as input. It replaces the original YAML front matter with the modified YAML data and returns the updated file content.

def recompose_qmd(yaml_data, original_yaml, content):
    # Convert the modified YAML data to string
    new_yaml_str = yaml.dump(yaml_data, default_flow_style=False)

    # Replace the original YAML content with the new one
    updated_content = content.replace(original_yaml, new_yaml_str)
    return updated_content

The workhorse of this code is the part that generates a summary of the Markdown content. I used the OpenAI chat API to do this. I created a chat prompt that asks the user to describe what the reader will learn after reading the article. I then used the gpt-4 model to generate a response to this prompt. The response is the summary of the article.

In my first attempts, GPT was starting the summary with “After reading the markdown article, the reader will learn”. I didn’t want this, so I added some text and an example to the prompt to remove this behavior.

def summarize_markdown(markdown):
    # Load your OpenAI API key

    client = OpenAI(
        api_key = os.getenv('OPENAI_API_KEY')
    )

    chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": f"""Without repeating 'After reading the markdown article, the reader will learn', 
            describe in one sentence what the reader will learn about after reading the markdown article. 
            Also don't say 'the reader will learn about', just say what they'll learn. For example, 
            instead of saying 'the reader will learn about strategies for programming', 
            just say 'strategies for programming' 
            Markdown:\n
            {markdown}""",
        }
    ],
    model="gpt-4")

    return chat_completion.choices[0].message.content

Finally, I wrote a function that processes all the .qmd files in a directory. It loops through all the files in the directory and calls the functions I wrote above to extract the YAML front matter, generate a summary of the Markdown content, and add the summary to the YAML front matter.

def process_qmd_files(directory):
    # Loop through all files in the directory
    for filename in os.listdir(directory):
        if filename.endswith('.qmd'):
            print(f'Processing {filename}...')

            file_path = os.path.join(directory, filename)

            # Read the file
            with open(file_path, 'r', encoding='utf-8') as file:
                content = file.read()

            original_yaml, full_content = parse_qmd(content)
            markdown = full_content.replace(original_yaml, '')

            if original_yaml is None:
                print(f'No YAML front matter found in {filename}, skipping...')
                continue  # Skip files without YAML front matter

            yaml_data = yaml.safe_load(original_yaml)

            if 'description' in yaml_data:
                print(f'YAML front matter already contains a description, skipping...')
                continue

            # Generate a summary of the Markdown content
            summary = summarize_markdown(markdown)

            # Add the summary to the YAML front matter
            yaml_data['description'] = summary
            
            # Recompose the file content
            updated_content = recompose_qmd(yaml_data, original_yaml, full_content)

            # Write the updated content back to the file
            with open(file_path, 'w', encoding='utf-8') as file:
                file.write(updated_content)

load_dotenv()

# Specify the directory containing your .qmd files
directory = <YOUR_DIRECTORY>

# Process all .qmd files in the directory
process_qmd_files(directory)

My blog has over 100 documents, and it took less than 2 minutes to add descriptions to all the fields. Using GPT-4 cost me $2.41, but if I was going to do this manually it will have taken me hours. I think it was worth it.

I used my ChatGPT Plus to generate the main block of code that has the function process_qmd_files, so the whole process, end-to-end, took less than 10 minutes.

Reviewers wanted

Packt Publishing is looking for technical reviewers to help with the book.

Reviewers need to have some programming experience and familiarity with AI. I don’t know exactly how large is the workload and what are the deadlines, but the book will have between 200 and 250 pages.

Reviewers get:

1. Credit in the book: name and bio printed on the front matter as a Technical Reviewer.
2. A complimentary print copy of the book at their doorstep.
3. Free 12-month subscription to all Packt books.

This can be a cool way of learning about the Semantic Kernel and AI, while also getting something for it.

If you are interested, please fill the form below and someone from Packt will contact you.

Why build this?

In 2021, I was writing a lot on LinkedIn, but I was limited by LinkedIn’s post formatting being mostly text — no inline images, no code. To have images and code, I started writing more on my blog, but usually finished my posts in the middle of the night, and I wanted to schedule them to post the next morning. I also wanted to be able to post to multiple social media sites at once. I created a simple app to post to LinkedIn, Twitter and Mastodon, and it worked well enough for my needs.

Now, in 2024, I’m using Threads a lot more, and an API is probably going to be available in 2024. I want to be able to schedule my blog posts to post to Threads, and I also want to share my journey learning the Threads API.

Technology Choices

Since I work at Microsoft, I am more familiar with Azure than other cloud providers. And since I work in data science, I am more familiar with Python.

My intended technology stack is:

• Azure App Service to host the app
• Azure Cosmos DB to host the data
• Azure Functions for the scheduled job (using Python)
• Python Flask for the web app, with Bootstrap helping make the UI better
• Quarto for the blog posts
• GitHub Pages to hold the blog posts

Project steps

The list below shows the steps I am taking to build this app. I will update this list as I make progress.

• Create an Azure Web App with Authentication and Basic Authorization
• Create a user database in Cosmos DB and link it to authentication
• Create a user profile page
• Connect to the LinkedIn API
• Enable a basic text post to LinkedIn

What problems I want to solve?

These days, I can take handwritten notes in most of my devices. My main device is a Surface Laptop Studio 2, which folds into a tablet and has a stylus. I also have an iPad Pro 13” with a stylus, but my Surface mostly replaced it. However, I cannot do everything with my Surface Laptop Studio 2. Here are a few of its problems:

• It is a little on the heavy side. It is heavier than the iPad Pro 13”, that many people already find too heavy for handwritten notes.
• Battery life is not great. I can get about 2-3 hours of battery life out of it, which is not enough for a full day of meetings.
• It is easy to get distracted by notifications and other apps on the device.

How does the reMarkable 2 solve these problems?

The reMarkable 2 solves these problems in the following ways:

• The reMarkable 2 is very light. It is 403 grams (0.9 pounds) and 4.7 mm (0.2 inches) thick. For comparison, my iPad Pro 13” is 682 grams (1.5 pounds) and 5.9 mm (0.23 inches) thick.
• It has no distractions. It is designed to be used for reading and writing, and it does not have a web browser or any other apps. It is very easy to focus on what you are doing.
• It has a very long battery life. I find myself charging it about once or twice a week depending on how much I use it. It charges via USB-C, which is easily available.

What is the reMarkable 2 useful for?

I have been using the reMarkable 2 for the following use cases:

• Academic paper reading and annotation: This was my intended use case for the reMarkable 2. The reMarkable 2 works, but it’s not ideal. It has a 10.3” screen, smaller than A4 paper, so I find myself reading papers in landscape mode, which makes the text a little bigger, but requires scrolling. In addition, I don’t like the contrast of the reMarkable’s e-ink display very much. It works, but it doesn’t have contrast adjustments, and it’s not as good as the Kindle’s display.
• Journaling: I have been using the reMarkable 2 for journaling almost every day. The writing experience is exceptional. It really feels like writing on paper, and the experience is very smooth.
• Note-taking in meetings: I have taken the reMarkable 2 to meetings, and this allows me to focus a lot better on meetings. The note-taking is superb, and the lack of distractions is great. The apps in the phone and in the PC allows me to easily see and add to my notes.
• Writing down ideas: I have been using the reMarkable 2 for writing down ideas and think about projects. It works well for that because it allows me to capture my ideas in a way that is easy to share and add to them later, without being distracted by technologies. This is specifically useful when I am in a plane or in a bus commuting.
• Many templates: the reMarkable has many paper templates, including lined paper, grid paper, and even music sheets. You can even create your own templates. I used to have more than one notebook for different purposes, and now I can have them all in the same device.

Downsides

The reMarkable 2 is not perfect. Here are some of the downsides:

• The reMarkable 2 is expensive, starting at $299 for the barebones model. You need a pen, and you probably want a cover. You can save money by buying unnoficial pens and covers at Amazon. One of my colleagues likes a $20 pen he bought online. The official pen costs $79 without an eraser and $129 with one. Covers run between $79 (cloth) to $199 (leather, with a keyboard).

• You cannot sync Kindle e-books to it. You can read PDFs and e-books on it, but you cannot sync your Kindle library to it.

• Its 10” screen is a little small for reading PDFs. I have been reading a lot of academic papers on it, and it’s easier to do it on landscape mode, where it gets a little bigger, but requires scrolling.

• The lack of contrast is a little annoying. The reMarkable 2 does not have contrast adjustments, and it’s not as good as the Kindle’s display. I find myself looking for better lighting more than I expected.

Do I recommend it?

Although I’m moderately happy with my reMarkable 2, it’s not for everyone and not for every use case.

It replaces my Moleskines and loose paper sheets, but it does not replace my iPad Pro completely: reading Kindle books and academic papers is still better on the iPad Pro.

Reading online reviews, people with ADHD seem to like the reMarkable 2 a lot, because it allows them to focus on what they are doing without distractions. I am not sure yet whether I have ADHD or not, but I likely do, and this may be biasing my opinion towards it.

Conclusion

The reMarkable 2 is, to me, a good replacement for loose paper sheets and Moleskines. A good Moleskine costs $30, so the reMarkable 2, at $299, pays for itself after 10 Moleskines, which is not that far off. It also helps me be more organized, because I can have all my notes in one place, they are always with me and I can easily search them and add to them.

The downsides are still problematic, and it may be better to wait for the next version of the reMarkable. It (barely) works for my use cases, but it may not be for everyone.

You can buy the reMarkable 2 at Amazon.

Explaining why memes are funny

In this first example, I will use the plugin to explain why a meme is funny. My code instantiates and calls the Vision plugin. The plugin has a single function, ApplyPromptToImage, which takes a prompt and an image URL as input, and returns the generated text. In my example, the prompt is simply Why is this meme funny?, and I pass an URL of an image of a meme.

    
    kernel = sk.Kernel()
    vision = kernel.import_skill(Vision())

    meme_base_url = "https://raw.githubusercontent.com/lucas-a-meyer/lucas-a-meyer.github.io/main/images/"
    meme_url_list = ["meme1.jpg", "meme2.png", "meme3.jpg"] 

    for url in meme_url_list:
        variables = sk.ContextVariables()
        variables['prompt'] = "Why is this meme funny?"
        variables['url'] = meme_base_url + url
        meme = await kernel.run_async(vision['ApplyPromptToImage'], input_vars=variables)
        print(f"{meme}\n\n")     

Here are the results. The Vision API does a good job of understanding the image and generating a good explanation of why the meme is funny.

Meme	Explanation
	The humor in this meme comes from the irony and self-awareness of the situation. The person in the image is depicted as being awake at 3 AM, a time typically associated with sleep, while reading about the importance of sleep. The implication is that while they understand the value of sleep for health and well-being, they are not following this advice themselves. The relatable contradiction between what they’re doing (staying up late) and what they’re reading about (the importance of sleep) creates a humorous juxtaposition that many people can identify with, as it’s common for individuals to do things that go against their own better judgment or knowledge.
	This meme is humorous because it presents a situation where someone has comically circumvented a rule through a technicality. The text explains that while there’s a rule against having pumpkins in their dormitory, there’s no mention of pineapples. The images show a pineapple carved in a manner similar to a jack-o’-lantern, traditionally made from a pumpkin, often for Halloween. The humor arises from the person’s literal interpretation of the rules, exploiting the lack of specificity to create a loophole and do something in the spirit of what’s prohibited yet not technically against the rules. This kind of wit and playful rebellion against overly specific or seemingly arbitrary rules often resonates with people who enjoy clever ways to bend or skirt the rules without breaking them.
	The humor in this meme arises from the juxtaposition of an inspirational saying typically found in areas of the home where food is prepared or consumed—like the kitchen—and placing it in a completely inappropriate context—the bathroom. Specifically, the framed quote says, “LIFE IS SHORT LICK THE BOWL,” which in a culinary context encourages people to enjoy life to the fullest, perhaps by savoring every last bit of their food. However, when placed in the bathroom, right next to a toilet bowl, the phrase takes on a literal and unpleasantly comical meaning. The idea of licking a toilet bowl is both absurd and humorous, evoking a reaction due to the drastic shift in the context of the phrase.

In the next section, I’ll explain how to use the plugin to identify an animal in an image and pass the results to another function.

Chaining the plugin answer to another function

In the example below, I first use the Vision plugin to identify an animal, and then use the results to ask for three interesting facts about an animal. I have createad a semantic function with the prompt Provide three interesting and unusual facts about the animal {{$input}} and called it AnimalFacts.

The URL I’m passing in the code below is a picture of an owl. The Vision API correctly identifies the animal as an owl, and the AnimalFacts function generates three interesting facts about owls.

url = "https://l.meyerperin.com/skv_owl"
gpt35 = OpenAIChatCompletion("gpt-3.5-turbo", api_key, org_id)
kernel.add_chat_service("gpt35", gpt35)

vision = kernel.import_skill(Vision())
plugins = kernel.import_semantic_skill_from_directory(".", "plugins")

variables = sk.ContextVariables()
variables['prompt'] = "What animal is this? Please respond in one word."
variables['url'] = url

animal = await kernel.run_async(vision['ApplyPromptToImage'], input_vars=variables)
facts = await kernel.run_async(plugins['AnimalFacts'], input_str=str(animal))
print(f"The animal from the picture is a {animal}")  
print(f"{facts}\n\n")

From the call above, I got the following result about owls:

1. Owls have specialized feathers with fringes of varying softness that help muffle 
sound when they fly. Their broad wings and light bodies also make them practically 
silent fliers, which helps them stalk prey more easily.

2. Unlike most birds, owls have both eyes facing forward which gives them better
 depth perception for hunting.

3. Some species of owls, such as the Great Gray Owl, can hear a mouse moving 
underneath a foot of snow from up to 60 feet away. Their ears are asymmetrical,
 with one ear being higher than the other, which helps them locate sounds in 
 multiple dimensions.

How to write native plugin to wrap the OpenAI API

In this section, I’ll explain all the steps involved in writing a Semantic Kernel native Python plugin to wrap the OpenAI API. I’ll use the Vision API as an example. The plugin code is in the GitHub repository.

from dotenv import load_dotenv
from openai import OpenAI
from semantic_kernel.skill_definition import sk_function, sk_function_context_parameter
from semantic_kernel.orchestration.sk_context import SKContext
import os

We need to import the OpenAI library, the SKContext class because our function has multiple parameters (the prompt and the image URL), and the decorators in skill_definition that will allow us to make the Python function visible to the semantic Kernel.

class Vision:
    @sk_function(
        description="""Asks the GPT-4 Vision API to perform an operation described by the prompt
        on an image given its url""",
        name="ApplyPromptToImage"
    )
    @sk_function_context_parameter(
        name="prompt",
        description="The prompt you want to send to the Vision API",
    )
    @sk_function_context_parameter(
        name="url",
        description="",
    )

The main code of the function consists in assembling a message that conforms to the Vision API specification. The message is then sent to the API, and the result is parsed and returned to the caller.

        def ApplyPromptToImage(self, context: SKContext) -> str:
        load_dotenv()
        client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

        response = client.chat.completions.create(
        model="gpt-4-vision-preview",
        messages=[
            {"role": "user", "content": [
                {"type": "text", "text": f"{context['prompt']}"},
                {"type": "image_url","image_url": {"url": f"{context['url']}",
            },},],}],
        max_tokens=300,
        )

        return response.choices[0].message.content

The full code for the plugin and associated demo can be found in the GitHub repository.

Have I ever thought about creating an app that does this as a service?

Yes, and with the blog post above, you can easily create one yourself. It will probably take you less than an hour to get it working. The main challenge is to bill for it, as the Open AI Vision calls can get expensive very quickly.

Introduction

The Microsoft Semantic Kernel is a thin, open-source, software development toolkit that makes it easier for applications to interact with AI services.

It was originally designed to power the Microsoft Copilots, such as Microsoft 365 and Bing. The initial version was in C#, but it has now been extended to Python and Java, and released to the developer community as an open-source package.

A whirlwind tour through Semantic Kernel in Python

In the post below, I’ll quickly show how to get started with Semantic Kernel in Python using an Azure subscription.

import semantic_kernel as sk
kernel = sk.Kernel()

from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion

gpt35 = AzureChatCompletion(deployment_name="gpt35", # yours may be different
endpoint=OPENAI_ENDPOINT,
api_key=OPENAI_API_KEY)

gpt4 = AzureChatCompletion(
    deployment_name="gpt4", # yours may be different
endpoint=OPENAI_ENDPOINT,
api_key=OPENAI_API_KEY)

kernel.add_chat_service("gpt35", gpt35)
kernel.add_chat_service("gpt4", gpt4)

prompt = """knock, knock? Who’s there? {{$input}}. {{$input}} who?"""

knock = kernel.create_semantic_function(prompt, temperature=0.8)

response = knock(“Dishes")
print(response)

import requests
from PIL import Image
import timm
from timm.data.imagenet_info import ImageNetInfo
from semantic_kernel.skill_definition import sk_function
from semantic_kernel.orchestration.sk_context import SKContext

class ImageClassifierPlugin:
    def __init__(self):
        self.model = timm.create_model("convnext_tiny.in12k_ft_in1k", pretrained=True)
        self.model.eval()
        data_config = timm.data.resolve_model_data_config(self.model)
        self.transforms = timm.data.create_transform(**data_config, is_training=False)
        self.imagenet_info = ImageNetInfo()

    @sk_function(
        description="Takes a url as an input and classifies the image",
        name="classify_image",
        input_description="The url of the image to classify",
    )
    def classify_image(self, url: str) -> str:
        image = self.download_image(url)
        pred = self.model(self.transforms(image)[None])
        cls = self.imagenet_info.index_to_description(pred.argmax())
        
        return cls.split(",")[0]
        
    def download_image(self, url):
        return Image.open(requests.get(url, stream=True).raw).convert("RGB")

classify_plugin = kernel.import_skill(image_classifier, skill_name="classify_image")

answer = classify_plugin.classify_image("https://links.meyerperin.com/tiger_jpg")
print(answer)

Plug-ins

One of the greatest strengths of the Microsoft Semantic Kernel is the ability of creating plugins. Plugins are collections of functions that can be imported into the kernel.

A semantic plugin is a collection of semantic functions. Each function should be in its own directory. Each directory should have two files: config.json and skprompt.txt.

The config.json file contains the configuration of the semantic function: which is the preferred engine to use, the temperature, etc.

The skprompt.txt file contains the prompt of the semantic function. The prompt is the text that will be sent to the engine to generate the response.

└───plugins
    └───jokes
        ├───cross_the_road_joke
        |   ├───config.json
        |   └───skprompt.txt
        └───knock_knock_joke
            ├───config.json
            └───skprompt.txt

{
    "schema": 1,
    "type": "completion",
    "description": "Generates a knock-knock joke based on user input",
    "default_services": ["gpt35"],
    "completion": {
        "temperature": 0.8,
        "number_of_responses": 1,
        "top_p": 1,
        "max_tokens": 4000,
        "presence_penalty": 0.0,
        "frequency_penalty": 0.0
       },
       "input": {
     "parameters": [{
      "name": "input",
      "description": "The topic that the joke should be written about",
      "defaultValue": "Dishes"
     }]
    }
}

knock, knock? 
Who's there? 
{{$input}}. 
{{$input}} who?


Repeat the whole setup and finish the joke.

jokes_plugin = kernel.import_semantic_skill_from_directory("plugins", "jokes")

knock = jokes_plugin["knock_knock_joke"]
print (knock("Dishes"))
# Dishes the police, open up!

print(jokes_plugin["cross_the_road_joke"]("Tiger"))
# Why did the tiger cross the road? To show the chicken it can be done.

Calling multiple functions in sequence

As you saw before, we loaded the native function classify_image into the kernel. And in the block above, we loaded the jokes plugin into the kernel. We can call both functions in sequence with the code below:

context = kernel.create_new_context()
context["input"] = url

response = await kernel.run_async(
    classify_plugin["classify_image"],
    jokes_plugin["cross_the_road_joke"],
    input_context=context
)

print(response)

This code will call the first (native) function that classifies an image, and then call the second (semantic) function that generates a joke about the image. Since we loaded the image of a tiger, the response will be something like:

Why did the tiger cross the road? To show the chicken it could be done.

Conclusion

I hope this brief tour helps you get started. If you want to learn more, you can check the official documentation.

You can also join the community Discord server.

Introduction

Midjourney is a generative AI tool that can generate high quality images from text prompts. It is particularly good at generating photorealistic images, and also good at blending an existing picture with a prompt. Lots of people use Midjourney as a digital diary, or to quickly create images for their social media and blog posts.

One unique characteristic of Midjourney is that its interface is only available on Discord, a popular chat app. In order to create images, you have to chat with the Midjourney bot on a Discord server.

This is a tutorial on how to get started with Midjourney. While it’s mostly based on the official Midjourney getting started guide, I have added a few suggestions that made my life much easier, especially my Step 2. Hopefully will help you, too.

Installing Midjourney

Basic usage of Midjourney

Now that Midjourney is installed on your server, you can start using it. I just send direct messages to it by right-clicking it and selecting Message.

The official guide has a very good introduction to the basic usage of Midjourney, starting on step 5. I will just summarize the main command imagine here, and show you an example of how you can use your own image as a starting point.

Conclusion

Google Duplex was going to change the world five years ago

Measuring screener performance

I wrote before about the many steps in an interview process. The last few steps of the process are: technical phone screen, on-site interview (now done remotely as well), and offer.

A technical screener can have a good idea of their performance by observing the results of the on-site interview step. If the screener is too lenient, lots of people will pass the phone screen but fail the on-site interview. The opposite is not necessarily a good sign. If the pass rate on the phone screen is low but the pass rate on the on-site interview is high, it could mean that the screener is too strict, but it could also mean that the candidate pool is bad.

Tech screeners should usually be more experienced than the average interviewer. Having a screener that is too lenient is costly, as many people will talk to a candidate that passes a technical phone screen.

The problem with LeetCoding

Some people use sites like LeetCode, StrataScratch and HackerRank to learn, but many use them to “grind” and simply memorize algorithms for interviews. Now that interviews are remote, some people cheat and copy the answers from these sites during the interview. The problem I need to solve is how to separate “grinders” (and cheaters) from people who know how to code.

My solution is imperfect, and may not work if the candidate is experienced, practiced a lot, but is nervous. I try to work around that problem in different ways. On the other hand, I think my solution worked in some important cases.

The solution

What I do is that I get some popular questions and change their assumptions to make them a lot simpler. During the intervew, I warn candidates that I did that. Even with the warning, the “grinders” regurgitate the solution from the websites, which don’t even solve the simplified problem. Good coders solve the simplified problem, and when I reintroduce the harder assumptions, they solve that one as well.

The Trap

It’s usually not hard to find what’s required to progress on your current job. You might have access to “career stage profile” documents that describe the characteristics of that job for different levels and what’s required to move from one level to another. Your colleagues or your manager can tell you what it takes to get promoted. Getting promoted is a good thing: it will give you a better salary and sometimes an expanded scope that will allow you to have opportunities to grow even further.

I had many career conversations in my tenure, and in a large proportion of cases, “getting promoted on the job I currently hold” is not a typical lifelong objective, but it’s typically an objective that people have a plan for. On the other hand, people frequently tell me about major goals/objectives (change professions, open their own company, get a Ph.D., etc.) for which they don’t have an action plan.

It’s like their algorithm is stuck near a local maxima (in this text, optimizing will mean maximizing) and the only thing they can see is the short-term objective, so they can only trace paths to get to the local maxima.

Eventually, people reach a local maxima and realize it’s not their global objective. They may not know what their global objective is, they just know “it’s not this”. Interestingly, the way that people solve that problem is not that different from what an algorithm would do if it realized it got stuck in a local maxima: take a big step, and start again from someplace else.

A simple fix

The more you know about the neighborhood about your global maxima, the easier it will be go trace a path to get there. Even if the path is not exact, at least you can tell whether you’re getting closer or further away from that goal.

Career coaches (and good managers) keep asking you about your “global objective”. By learning more about it, they can tell you more the path to get there, or find someone that can. Over time, you may accumulate enough information to make a better move when you decide to.

The multi-armed bandit

The hardest problem to solve in career optimization is lack of information. People may have some general idea of where they want to go, but usually not much. For example, I commonly hear people saying that they want to become managers, but after just a few questions I realize that they don’t really know what that means and whether they actually want that.

One way to maximize happiness under incomplete information is to explore several different things, and once you find what makes you happy, then you exploit it (although “exploit” is a somewhat aggressive word in English, it’s actually the technical term), getting as much happiness as you can from it. There are algorithms to find a good balance between exploration and exploitation. One example is the “multi-armed bandit”.

The concept comes from slot machines in casinos. Because a slot machine has one lever that people pull to activate it and people usually lose money on it, it’s called an “one-armed bandit”.

Imagine a slot machine that has multiple levers, therefore a “multi-armed bandit”. When pulled, each lever gives a prize based on a statistical distribution that is unknown to the player. You pull the first lever and get $100. Should you pull the same lever again or switch? Maybe one of the other levers has a constant payoff of $1,000. Maybe that lever you just pulled is actually the best one.

Without knowing the distributions, you can’t know the optimal action to take. In real life, not only one doesn’t know the distributions, but the distributions for each lever can change over time. If that was not complicated enough, you also have infinite levers to choose from. Nobody said it was supposed to be easy.

The multi-armed bandit problem doesn’t have a general optimal solution, but has several strategies that achieve approximate solutions. It is a classical reinforcement learning problem.

Many strategies are very simple “epsilon” strategies, for which one decides a proportion epsilon (between 0 and 1) of exploration (trying to figure out the distributions of the available actions), and the remainder of the time is used for exploitation (performing the action that is expected to yield the best results).

In career conversations, I always suggest that people take the time to explore new things, especially things that are in the (conceptual) neighborhood of what they think they want. Because I’m older than the average tech person, they may think this is some form of ancient wisdom, but I’m just pointing them towards a classical (and somewhat modern) reinforcement learning algorithm.

That’s using data science in production.

The discipline of writing about what you’re doing

When I was a wee little kid and had just entered college, one of my first classes was “Physics Lab”, and the first class of that was to measure “gravity”, more specifically, the gravitational acceleration . Of course, mathy Computer Science studies that we were, we all knew that the gravitational acceleration would be .

Measuring it, however, it’s not very easy. First, remember that this is in the early 90s in Brazil, so digital cameras were very rare. Part of the problem is that 9.8 meters (approximately 30 feet) is quite high, so if we wanted our experiment to take around 1s, we would need a big ladder. Or, as it happened, we’d need to run an experiment that took less than a second for each run. In a class with 25 students, that was the preferred route.

We set up a vertical track attached to a device that would spark every 1/60 of a second, and we attached grid paper to the track. This is called a Behr free fall apparatus.

We would release the device from the top of the track, and the sparks would mark the grid paper. We would then manually measure the distances between the sparks and the differences between the distances would tell us the acceleration. In theory. Again, remember this is the early 90s, so there’s no Windows 95 or Excel easily available, there were several steps that were prone to error.

The desired outcome was not only to calculate the acceleration due to gravity, but also to generate a lab report.

Writing the lab report

It was a simple experiment, but there was a twist. Another class of 25 students would have to replicate the experiment following the lab report from the first 25. Oh boy. We quickly found out that the best lab reports were the ones in which the experimenter would document the experiment as they were executing the experiment. Another thing that worked was going through the experiment more than once. What did not work was to perform the experiment and then go to where the computers were to type up a report from memory.

I thought that was very insightful, shortly afterwards, we would learn about Donald Knuth’s proposed paradigm of literate programming, which lives on in Jupyter Notebooks and R Markdown and that is very popular in data science today. Behind it is the same concept of explanations interspersed with technical work to produce something that is reproducible and easy to understand.

Blogging helps you train to write about your work

Besides the excellent reasons offered by David Robinson for aspiring data scientists to blog, I think experienced data scientists can also benefit from blogging. David argues that aspiring data scientists should blog for technical practice, to build a portfolio, and to get feedback. Experienced data scientists might benefit for similar reasons, but not exactly the same.

Similar to “practicing”, blogging about technical topics helps you learn how to communicate better about them, and also helps you learn how to write better research reports. Similar to “portfolio”, it signals a data scientist’s body of work. For an early-in-career data scientist, it may be more showing that you can do analysis, but for an experienced data scientist is more about what type of analysis you like to do. Finally, I think that “feedback” also works differently for early-in-career and experienced data scientists. I’m not sure random strangers in the internet will sweep in and correct your posts, but you can find out what people are reading and sharing. You may be surprised when you learn what parts of your work people find interesting.

Should we care?

There’s a lot of information asymmetry in the Data Science market, the position being so new, and “Michael Spence signaling” matters. Having a “Data Scientist” title from a reputable company is an important signal. For example, a medium-sized company might not be aware that Lyft recently changed their data analysts title to “Data Scientist”

Part of the problem is that the “Data Scientist” title is itself poorly defined, as shown by the litany of Venn and not-Venn diagrams listed above. A short, non-comprehensive list of alternative titles for Data Scientists I’ve met in the last month:

• Tableau/Power BI Report Designer
• Machine Learning Researcher
• Python Programmer who knows how to call scikit-learn functions

Using the same name for vastly different abilities makes matching between employers and employees difficult. In addition, it may make career progression difficult. For example, job responsibilities and objectives are likely to be related to a company’s interpretation of the title of “Data Scientist”. Employees in that company may be “Data Scientists” under other interpretations, and may be really good, but may have their performance evaluated under different guidelines: “I know your ML model saved $1B for our company, but the color scheme of your Tableau reports is horrible!” As per the (not really) Albert Einstein’s quote “(…) if you judge a fish by its ability to climb a tree, it will live its whole life believing that it is stupid.”

Conway’s Venn to the rescue!

Here’s where I think Conway’s Venn diagram comes to the rescue in a useful way. We could rule out the Tableau Designer if they lack programming or statistical skills. We could rule out the ML Researcher if they don’t have domain expertise. We could rule out the Python programmer if they don’t know Math and/or have domain expertise. Perhaps this makes Data Scientists hard to find (it would justify their salary)!

Depending what the company you work for was before the Data Science profession started, a lot of people from different jobs may have re-branded themselves as data scientists. I think Conway’s diagram is helpful as it points to where people should focus their development in order to be more effective as data scientists. For example, former Economists most likely need more hacking skills, former computer scientists need more Statistics and domain expertise, and former management/technical consultants likely need more Statistics.

It can also serve as a check for inclusiveness. For example, a team of former machine learning engineers is struggling to deliver meaningful projects because they lack domain expertise. However, they can’t hire domain experts because the domain experts keep failing the machine learning interviews. Comically, the machine learning engineers think that domain expertise “is easy”.

The most useful part of Conway’s diagram for me is that I’ve time and again observed projects that don’t have representatives that can communicate well with each other in all three areas to fail. Conway’s diagram even makes it easy to show how the project failed (“nobody with domain expertise”, “missing knowledge of statistics”). Having a single person that can do all three is difficult, but it certainly solves the communication problem. That being hard or impossible, having specialists that can communicate with each other will usually be sufficient.

But Conway’s Venn is missing something!!!

True to the famous Box aphorism that “all models are wrong but some are useful”, Conway’s Venn is not perfect. In my experience, one of the things that I usually miss is “intensity”. Someone that knows how to write a “Hello World!”, knows how to take averages, and know how to calculate compound interest has knowledge in all three areas of the Venn diagram, but is probably not the ideal person to lead a data science project in Finance. Conway’s is best used to tell you what’s missing for career development, for a team, for a project, but it will not solve all problems.

It will, however, solve many problems that matter, and will provide useful information very elegantly. You can’t ask much more from a model.

Should data science teams be centralized or distributed?

I was happy to find that Jonathan Nolis has the same opinion as I do: that a data science team works best when distributed into client areas. In addition, we agree that if you distribute your data science team among several different client areas, you have to find a way of keeping the data scientists feeling as if they’re part of a cohesive group. And that’s hard.

For me, a key part of the argument to distribute data scientists in client areas relies on Drew Conway’s (ancient) Data Science Venn diagram.

Assigning data scientists to a client area will help build the “Substantive Expertise” part of the diagram. A common trap here is “Well, we can do the same if we keep the data scientists centralized and give them specializations that match the client areas.”

Although I think that can work, I worry about what happens when data scientists have downtime.