Programming with LLMs

Addressing the elephant in the room

This is the first of a two-part mini-series on programming with LLMs. It is for everyone, not just developers. Thanks to LLM's help, it is now easier than ever to be a “programmer,” with a few caveats.

In this first edition, we explore these caveats and share our opinions. On Thursday, we will share our best applicable tips for programming with LLMs.

P.S. we will share more tips with our premium supporters! Here’s a discount if you want to try it out:

According to the recently published 2024 Stack Overflow Survey:

76% of all respondents are using or are planning to use AI tools in their development process this year, an increase from last year (70%). Many more developers are currently using AI tools this year, too (62% vs. 44%).

What does this mean for the future of programming? What are the main benefits of generative AI, and how can LLMs safely accelerate software development?

The Elephant in the Room

Let’s address the elephant in the room: Will programmers still be needed?

Programmers aren't going anywhere soon. While most people will be able to program basic software, the demand for skilled developers will continue to grow as LLMs become more capable. We need experts when LLMs fail, and for tasks they have not yet been trained to do. They cannot generalize as well as humans can.

The role of a programmer is set to evolve dramatically. LLMs are becoming indispensable tools for most developers, transforming how code is written and projects are built.

Some developers still work in assembly, crafting new instructions for the latest generations of CPUs and GPUs. Security experts also rely on low-level assembly language. However, most developers no longer use assembly directly because we've built layers of abstraction to simplify the code.

Imagine having to write (in Python) :

print("hello world")

In assembly language, it would look like this :

section .data
    msg db 'hello world', 0xA  ; 'hello world' string with newline

section .text
    global _start

_start:
    ; write system call
    mov eax, 4        ; syscall number for sys_write
    mov ebx, 1        ; file descriptor 1 (stdout)
    mov ecx, msg      ; message to write
    mov edx, 12       ; message length
    int 0x80          ; interrupt to invoke system call

    ; exit system call
    mov eax, 1        ; syscall number for sys_exit
    xor ebx, ebx      ; exit code 0
    int 0x80          ; interrupt to invoke system call

*An LLM helped us generate that. We do not program in assembly. That’s the power of LLMs. (p.s. there might be errors in the code; Assembly programmers, please be comprehensive)

Not to mention ensuring it works across different CPUs and operating system versions.

LLMs represent a new level of abstraction, making it easier for developers to communicate with computers—just as high-level programming languages abstract away the complexities of assembly language (or Python, an abstraction of C). The only difference is that LLMs make mistakes. They work with statistics, not bits.

People Misunderstood Devin

Devin is the first attempt at automating the software development process—from initial requirements to building a plan, coding with integrated tools, and using the command line to interact with the environment. The name “Devin” was trending for good and bad reasons when it first came out. Some praised the project as it promised to solve programming once and for all. Others were skeptical of the possibility of creating a fully automated software development agent and for good reasons.

The media buzz was so loud that everyone missed the point.

Devin aims to develop an LLM-based system to tackle real-world GitHub issues (specific programming tasks). A dataset called SWE-Bench includes 2,294 software engineering problems sourced from real GitHub issues, paired with the commits that resolved them. As of September 2024, Aide leads the leaderboard by resolving 43% of these issues, while Devin currently ranks 18th, resolving 27%. Training LLMs to handle GitHub issues is a smart approach since LLMs excel at analyzing text and handling large datasets. Teams of researchers are working to improve these models' accuracy.

The key takeaway is that while LLMs will eventually resolve most issues, human input will remain essential.

Developers will still be needed to identify and define problems. The future of software development lies in a collaborative approach: humans define the requirements, and LLMs handle much of the coding, speeding up the development process.

Programmers using LLMs for coding will transition into more of a reviewer role. Some argue that LLMs are making us lazier. No. Those who are technically inclined and enjoy solving complex problems will continue to do so more efficiently with the assistance of LLMs.

💡Benchmarks are comparable to unit testing but for models. It’s the way to evaluate the performance of the LLM.

On Thursday, we will share the best real-world tips for programming with LLMs that we currently use daily. Stay tuned!