AI Regulation, Prompt Compression Improves Accuracy, & LLMs Learn from Mistakes
Stratechery's Ben Thompson gives a thoughtful take on AI regulation, Microsoft develops a prompt compression technique that improves LLM accuracy, and LLMs can learn from their mistakes
Misunderstood Innovation & AI Regulation
The biggest news this week in AI was President Biden's executive order on Safe, Secure, and Trustworthy Artificial Intelligence. A couple days after the dust settled, Ben Thompson of Stratechery wrote a thoughtful article about innovation and regulation by contrasting the mindsets of Bill Gates & Steve Jobs during the war over mobile.
Gates shows how even innovators get innovation wrong. He was so fixated on PCs as the computing hub, he failed to innovate the Windows Mobile interface. Jobs, on the other hand, was uncertain about the future, and that uncertainty led him and Apple to introduce a revolutionary multi-touch interface with the iPhone. The rest was history.
Ben extends Jobs’ and Apple’s success towards AI regulation. He suggests that innovators and regulators should follow Jobs’ lead, embracing uncertainty and the unknown potential of technology, rather than preemptively stifling it with restrictive regulations, echoing Microsoft’s missteps. He warns that the government’s current approach risks inhibiting the very innovation that could benefit humanity, while also providing regulatory capture for the current AI front runners.
When Microsoft made the wrong call on mobile, only they lost out. The rest of the world still received an incredible amount of consumer surplus through iOS and Android (which changed their OS from resembling Blackberry to iOS after the iPhone release).
For regulation, however, the wrong prediction could prevent all of society from realizing the consumer surplus of an innovation that never happens.
Microsoft Tackles LLM 'Lost in the Middle' Phenomenon with Smart Compression Technique
A recent discovery shed light on the 'Lost in the Middle' phenomenon affecting Large Language Models (LLMs), where crucial information is overlooked when positioned within the middle of lengthy contexts.
Microsoft's research team developed a a prompt compression strategy, termed "LongLLMLingua". The method involves two steps:
Coarse-grained reduction of the context based on document-level perplexity, followed by a fine-grained trimming using token perplexity.
Fine-grained compression of the remaining text via token perplexity
This approach enhances the model's performance, and reorganizes the context by placing the most significant passages at the beginning and end, which leverages the positional bias to the model's advantage. The result is a streamlined, efficient, and cost-effective process that transforms a potential weakness into a significant strength for LLMs.
LLMs Learn From Mistakes, Similar to Human Students
A new white paper, also with Microsoft participating, introduces “Learning from Mistakes” (LeMa), which teaches LLMs to solve math problems by learning from errors, similar to how humans learn by correcting their mistakes.
By replicating the error-driven learning process, LeMa fine-tunes LLMs on mistake-correction data pairs generated by GPT-4, significantly improving performance.
Their code, data, and models, are available on GitHub.