I’ve been writing PHP since 1996, and consider myself a competent developer. At various times, I’ve run some pretty popular open source projects and have written and released a gajillion lines of code, translated into 30+ languages, deployed all over the world.
Until a couple months ago, I hadn’t really tried to use AI to generate code. Software development seems to be among the leading use cases for large language models and there are a bunch of high valuation companies doing AI based code generators. I decided it was time to give vibe coding a real chance.
I decided that for this project, I would use AI to code an entire simple application. I think this use case best reflects the “AI is going to replace developers” discussion, so I wanted to test that. I was going to accept AI code, not be too picky, and see if I could get a functional PHP web application completed. I’m going to use multiple tools, including a paid version of ChatGPT and various locally installed LLMs for this project. In the next installment, I’ll talk about agentic coding tools like Codex. I will be posting the completed project to GitHub in the future. Stay tuned!
For now, I have come a LONG WAY and learned a lot. I feel like there are a few techniques in particular that made a huge difference in my ability to repeatably generate usable code. Read on!
What I Learned
LESSON 1: High cost of AI Tokens biases systems towards laziness.
What happens is that many of the popular cloud based AI systems will do things like include “placeholder” functions that are empty, they will truncate output, and similar actions to reduce the output size. Behind the scenes, you are provided with a “token budget” for your sessions. The system tries to keep you within your budget by taking shortcuts to reduce the size of your output. The result is missing code that you have to track down. The impact of this is detrimental for novice coders, because it will often quietly make these changes.
MITIGATION: This remains a fact of life for the time being.
Extremely explicit instructions somewhat mitigate this concern. Breaking your application into modules helps, but requires more advanced knowledge of the codebase. No matter how I ask the questions, however, this problem sneaks in. Eventually, as AI related costs come down, this will be come less of a probelm. For now, here are a few prompt additions that can help:
- Verify that you have followed all instructions and implemented all requested features. If you ignore any instructions, tell me what you ignored and why as part of the response.
- You may use the full token budget to output the complete code. Do not truncate.
- Do not use placeholders, skeletons, or stubs. Include the full implementation of all features. If you skip any feature, the answer is invalid.
LESSON 2: LLMs are much better at writing new code than updating existing code
Time and time again, I would provide code and ask for an update. Most of these systems use some flavor of pattern matching rather than analyzing the entire file. I think this goes back to costs. The result is that you can ask for a simple update 10+ times and it will never get it right. Along the way, it will create parse error after parse error. Things like putting PHP code as text into an HTML input text box.
Ultimately, LLMs are best at creating new content. Going back and updating existing content is not really what they were designed for. In fact, tasks involving updates are often done using regex, diffs, and similar tools.
MITIGATION: Don’t ask it to update code.
Even minor, simple updates can result in 10 iterations of parsing errors. This is the case with LLMs optimized for coding as well as general purpose. Eventually, you may successfully get your code generated, but the odds of losing other code, or wasting hours to implement a 5 minute change is very high. I end up pasting some phrases at the end of almost every prompt now, here are a few that may help:
- Open the file and analyze it in detail so you understand what the code does. Then, apply ONLY [your requested change] based on best practices. Do not simply pattern match. Do not change anything else. Provide the complete updated file.
- Open the file and manually verify that the code is well formed and will parse.
LESSON 3: Treat the prompt as the code.
Because of the previous lessons, we now know that AI cuts corners and isn’t well suited to working in reverse. The result is that it is difficult to start simple and iterate on a project to improve and extend it. This challenge is another expression of the fact that LLMs at their core begin life as token predictors. When you are updating code, you are out of that realm and into a land where tools like regex are required.
MITIGATION: Iterate on the prompt, not the code.
When you get an output that you like, freeze the prompt, not the code. Now, when you want to extend or enhance that output, update the prompt and regenerate the whole thing using your good prompt with whatever changes. Now, instead of using a version of your code as your “gold copy”, do that with your prompt. Basically, every time you iterate, run the whole prompt to generate the whole project again. Once you get to the point that your project is too big to do this, its time to move on to a purpose built coding tool like Codex.
My Take
Vibe Coding gets a lot of hate, especially from people who write code for a living, which makes complete sense. If a “CTO Robot” came out tomorrow, the first thing I’d do is look for the things it stinks at.
In the hands of a non-developer, these things can make really awful projects. But as another tool for a seasoned developer? Its an amazing productivity multiplier. For example, here is an excerpt from one of the MANY prompts I executed in the creation of this article:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
This archive contains a small PHP web app that manages native MySQL/MariaDB backups and restores. The current implementation constructs shell command strings (with redirection `>`, `<` and optional gzip pipes) and executes them via proc_open($cmdString). It also embeds secrets like MYSQL_PWD into the command string. Refactor ONLY the command execution portion to be safer and not rely on shell parsing, while losing NO functionality. Details: - Keep the same UI behavior, output filenames, log format, success/failure handling, and any existing return values/messages. - Replace shell redirection/piping with proc_open using argument arrays and PHP stream piping: - Do not use `sh -c`, Do not use `shell_exec`, Do not use backticks, Do not concatenate `>`/`<` or `|` into a command string. - Use proc_open with an argv array: ['mysqldump', '--host=...', ...] and descriptorspec to connect STDIN/STDOUT/STDERR. - For writing backups: connect mysqldump STDOUT to a file handle (for .sql) OR to a gzip process STDIN whose STDOUT is the .gz file handle. - For restoring: connect file handle (or gunzip process STDOUT) to mysql client STDIN. Implement new execution helpers in helpers.php: - proc_open_argv(array $argv, array $env = [], $stdin = null, $stdout = null, $stderr = null): returns ['exit_code'=>int,'stdout'=>string,'stderr'=>string] while supporting passing stream resources for stdin/stdout and capturing when needed. - run_pipeline(array $procs): ability to connect stdout of proc A to stdin of proc B using stream_copy_to_stream; avoid deadlocks; close appropriate pipe ends. Update backup/restore code paths to use the new safe functions: - Backup (.sql): mysqldump argv -> stdout file stream. - Backup (.sql.gz): mysqldump argv -> gzip proc stdin; gzip stdout -> .gz file stream. - Restore (.sql): file stream -> mysql argv stdin. - Restore (.sql.gz): gunzip (or gzip -dc) argv stdin from file stream; its stdout -> mysql argv stdin. Ensure no functionality regressions: - Preserve exit code semantics: if any stage in a pipeline fails, return nonzero and include stderr info. - Preserve “command preview/masked command” in logs. - Preserve compression option and filename conventions. |
If you take a second to read through that prompt, you’ll see that it would probably only make sense to another software developer. Someone just learning PHP or MySQL probably wouldn’t know what a lot of that means. I look at these tools the same way I look at any other. A skilled carpenter can do more with his toolbox than a beginner can, so he isn’t afraid of the beginner stealing his job.
In the next part, we will examine agentic coding along with some other approaches, topics, and tools.