Puttu Pattu

Fabrice Bellard: Portrait of a super-productive programmer (2011)

Fabrice Bellard: Portrait of a super-productive programmer (2011):

Fabrice Bellard: Portrait of a Super-Productive Programmer

Posted by Cameron Laird on Mon, Mar 21, 2011 @ 03:28 AM

Hollywood portrayals of computing superstars are more rooted in comic-book super-heroics than the realities of software development. Except that in programming, superpowers do exist. Fabrice Bellard has them.
Computing in the movies is hard to recognize: Typographical errors are non-existent, crackers break through tested defenses in seconds, and practitioners create twenty-function-point programs in fewer keystrokes than the count of bullets whizzing by their heads. Hollywood portrayals are more rooted in comic-book superheroics than the realities of software development.
Except that in programming, superpowers do exist. Fabrice Bellard has them.

Serial achiever

What other description can there be when one man successively produces a dozen important and widely-used software applications? Look at the (partial) record:

• 1997: Publication of Bellard’s formula for computation of digits of pi

• 2000: Calculation of the largest-known prime. (Make sure you read this one—the source code is only 438 bytes long.)

• 2001: A self-compiling compiler of a subset of C that fits in 3 kilobytes of memory

• 2004: TinyCC Boot Loader

• 2005: DVB-T computer-hosted transmitter

• 2009: et world record for calculation of the digits of pi

Every one of these would be a proud accomplishment in any programming career. To produce so many, and so consistently, is very, very rare.
A bit of background provides the context to understand this record clearly. It’s common in programming folklore to talk about how a particular program was pulled together in one intense weekend, or during a vacation interval. Many of these tales are true, as far as they go; several of the best-known programs were first launched with a specific effort of just a few days. This leads some to expect that those few days define the life cycle of program development: A sufficiently clever programmer can create the Linux kernel this holiday season, take the next weekend off to invent the Ruby programming language, build an award-winning iPad game the following month, and so on.
It’s not so. Just because top sprinters finish a hundred meters in under ten seconds doesn’t mean they sustain that pace for the duration of a marathon. When Bellard wrote LZEXE, for example, he created a program that performed the data compression he needed at one particular time (it was the first widely-used executable file compressor for MS-DOS).
What’s notable is that he also invested the follow-up effort to ensure the software was engineered well-enough to work on other computers and on multiple operating systems, could be understood and maintained by others, was adequately documented, licensed to be useful, and so on. All those apparently secondary activities typically  take orders of magnitude more time than the original coding at the heart of a successful program.
Bellard isn’t unique as much for being a smart guy who has clever ideas about programs—although he certainly is and does—but for his achievement in refining and crafting those ideas into programs that make sense to others. He consistently creates applications that other programmers want to use and enhance.
Think about QEMU, for example. Like all Bellard’s best-known programs, it’s freely-available under the GNU Public License (GPL) or a close relative, originally developed under Linux, broadly portable, and now mostly maintained by others. Several programs preceded QEMU and roughly fit its description as “a generic and open source machine emulator and virtualizer;” QEMU, though, like many of Bellard’s creations, combines performance, robustness, portability, and flexibility in a way that none of its ancestors or rivals matched. Bellard’s contribution wasn’t to create the idea of a machine emulator, but to envision how to engineer one well enough to move it from the classroom or laboratory to “working-class” developers’ and testers’ toolboxes. It’s now indispensable to many testing teams, who can leverage QEMU to manage hundreds of distinct computing environments from a single underlying physical computer.

Finding a balance

Bellard seems to balance nicely between extremes that threaten productivity. His techniques are modern without being faddish, and careful, yet not neurotic. While he moves every few years into new and fertile unconquered territory, he exercises  patterns that have served him well over and over: cleanly-styled C, data compression, numerical methods, signal processing, pertinent abstractions, media formats, open-source licensing, and “by-hand parsing.”
Bellard doesn’t appear to promote himself—he politely declined to be interviewed for this profile, for example—but others enthusiastically adopt his creations. Among the 654 Copyright notices QEMU 0.13.0 source code embeds, 216 are his. To put that another way: He was successful enough with QEMU during its first few years to entice others to put in over twice as much work (by this crude measure) as he has!

Bellard, born in 1972, began practicing his own coding techniques first on a TI-59 scientific calculator, at the beginning of the ‘80s. Many of the projects listed above, along with others in his portfolio, originated as student assignments at l'École Polytechnique, where he matriculated in 1990. TinyGL, for example, derives from the VReng Virtual Reality Engine distributed 3D application he began in 1998. Based on this experience, he eventually created and released in 2002 TinyGL as a very fast, very compact re-implementation of a substantial subset of OpenGL.
Similarly, his recent and relatively well-publicized calculation of several trillion (!) digits of pi was performed on a single desktop computer. Previous record-setting calculations of this class were done on million-dollar supercomputers. Bellard made it seem natural to pull together his mathematical insight, broad experience at instruction-level coding, and careful engineering to advance the field this way. The roots of the achievement reflect back to his experiments on a hand-held programmable calculator at the age of nine .
To describe fully even one of the projects already mentioned is beyond the scope of an article such as this; it would take several books to capture all that Bellard has already contributed, and at least another to explain how he works. A few broad conclusions are possible, though:

• As has often been observed before, the very best programmers are far, far more productive than the average, able to produce multiples of the usable output of others;

• While it’s unrealistic to predict how programming would look without Fabrice Bellard or open-source licensing, there can be no doubt that such programs as QEMU, LZEXE, and FFmpeg are in pervasive use thousands of times each day around the world now; and

• If there’s a secret to this superhero-level productivity, it appears to have less to do with comic-book mutation and radioactivity, and far more with discipline, confidence, rigor, and many years of practice.

Mass media make it easy, especially in the United States, to believe that only “clashes of the Titans” matter. That is, the mindset is that progress with computers has to do with million-dollar budgets, celebrity announcements, and courtroom jousting. That’s a dangerously narrow view, dangerous because it neglects the creativity and insight of sufficiently dedicated individuals. While Fabrice Bellard seems unlikely ever to hire a publicist or otherwise elevate his profile in the public eye, he shapes our use of computers more indelibly than corporate executives and industry analysts.
How do you use FFmpeg or QEMU or his other productions?  Tell me about it in the comments.  Even more important: What great program will you write next?  What obstacles to superproductivity do you face?
See also:

• Top 5 Developer Skills That Will Get You Hired or Promoted
• What Wayland Means for Developers
• 14 Ways to Contribute to Open Source without Being a Programming Genius or a Rock Star

Siebel – Useful Business Services

BS NAME	METHOD	DESCRIPTION	WHERE TO USE
FS Holiday API Service	GetElapsedBusinessTime	Returns business time taken to accomplish task.	It is used when you want to work on some date or date related information
	GetResponseTime	Returns commited time based on service calander and exception calander
	IsHoliday	Returns TRUE or FALSE if given day is holiday and return next working day
Workflow Process Manager	RunProcess	Used to run a workflow	Used either explicitly or internally to run a workflow
SLM Save List Service	LoadPopupApplet	Used for displaying a pop applet at run time. Details-The Applet Used should have type standard and contain the class as  CSSSWEFrameListPopup. If you are facing any problem creating the applet then just copy the vanilla applet - FINS Service Request Order ID Popup Applet and then customize it according  to the requirement	To show a pop applet at run time
ISS Add Attachment Service	AddAttachment	AttachmentFieldName (Lit)( AccntFileName)BusinessComponent(Lit)( Account Attachment)BusinessObject(Lit)(Account)File(Lit)(Path of the attch)HeaderBusinessComponent(Lit)(Account)ObjectIdHere – Lit – means – Literal – OR – hard coded value.AttachmentFieldName – is the name of the field on which we want to add the attachment. In our case it is AccntFileNameBusinessComponent – Name of the Business Component to which the field belongs. In our case we have chosen - Account Attachment – as the Business Component.BusinessObject - Name of the Business Object to which the chosen Business Component belongs. In our case we have chosen - Account Attachment – as the Business Component. In this case Business Object is Account.Object Id – Row id of the record.All these are input arguements to the method and there is NO Output arguement from the method.	Used to add the attachment when required to any of the business component
SIS OM PMT Service	Set Profile Attribute	Used to set and get the Profile Attribute	Profile Attributes are used For Global purpose
	Get Profile Attribute
FINS Goto View Service	GotoView	Used to navigate to different view	When you want to navigate to different view
FINS Teller UI Navigation	RefreshCurrentApplet	Used to refresh the current applet	Used when we want to refresh the current applet without breaking the sequence
	GotoView	Used to go to a view you want	Nevigate to a view you want
FS Verify Entitlement	GetBestResponseTime	If you want to get the response time from entitlement matrix by passing  the priority and the entitlement id  and entitlement name as input but it will not take calendar into consideration. It will give calendar id as an output.	Used in service request to get the committed time based on the entitlement matrix. It is one of the block in the Vanila  workflow FS – Verify Entitlement SR.
Data Validation Manager	Validate	Used to Validate a rule set  present in Administration-Data Validation.This returns the result code and result message which means the message in the validation messages - associated with the rule set and can work by passing the rule set name only.It can be called from either from workflow or from the run time event	Scenario- In an insurance company, claim adjusters are required to enter a closed date whenever they close a claim. If the adjuster tries to close a claim without a closed date, an error message appears and the claim record is not committed to the database.
SiebelUserProfileService	GetProfileAttribute	Used to set the profile attribute and get the profile attribute	To set and get profile attribute

The Flowchart Of Gandalf’s Problem-Solving

The Flowchart Of Gandalf’s Problem-Solving:

He’s a wizard so you’d think he could solve just about any problem all on his own, but even he needs help from a few select sources. This flowchart created by Emil Johansson of the Lord of the Rings Project breaks down Gandalf’s choices and how Hobbits and eagles figure prominently in every plan of attack.
(I Love Charts via Laughing Squid)

The Unreasonable Effectiveness of C

The Unreasonable Effectiveness of C:

The Unreasonable Effectiveness of C

For years I've tried my damnedest to get away from C. Too simple, too many details to manage, too old and crufty, too low level. I've had intense and torrid love affairs with Java, C++, and Erlang. I've built things I'm proud of with all of them, and yet each has broken my heart. They've made promises they couldn't keep, created cultures that focus on the wrong things, and made devastating tradeoffs that eventually make you suffer painfully. And I keep crawling back to C.
C is the total package. It is the only language that's highly productive, extremely fast, has great tooling everywhere, a large community, a highly professional culture, and is truly honest about its tradeoffs.
Other languages can get you to a working state faster, but in the long run, when performance and reliability are important, C will save you time and headaches. I'm painfully learning that lesson once again.
Simple and Expressive
C is a fantastic high level language. I'll repeat that. C is a fantastic high level language. It's not as high level as Java or C#, and certainly no where near as high level as Erlang, Python, or Javascript. But it's as high level as C++, and far far simpler. Sure C++ offers more abstraction, but it doesn't present a high level of abstraction away from C. With C++ you still have to know everything you knew in C, plus a bunch of other ridiculous shit.

"When someone says: 'I want a programming language in which I need only say what I wish done', give him a lollipop."
- Alan J. Perlis

That we have a hard time thinking of lower level languages we'd use instead of C isn't because C is low level. It's because C is so damn successful as an abstraction over the underlying machine and making that high level, it's made most low level languages irrelevant. C is that good at what it does.
The syntax and semantics of C is amazingly powerful and expressive. It makes it easy to reason about high level algorithms and low level hardware at the same time. Its semantics are so simple and the syntax so powerful it lowers the cognitive load substantially, letting the programmer focus on what's important.
It's blown everything else away to the point it's moved the bar and redefined what we think of as a low level language. That's damn impressive.
Simpler Code, Simpler Types
C is a weak, statically typed language and its type system is quite simple. Unlike C++ or Java, you don't have classes where you define all sorts of new runtime behaviors of types. You are pretty much limited to structs and unions and all callers must be very explicit about how they use the types, callers get very little for free.

"You wanted a banana but what you got was a gorilla holding the banana and the entire jungle."
- Joe Armstrong

What sounds like a weakness ends up being a virtue: the "surface area" of C APIs tend to be simple and small. Instead of massive frameworks, there is a strong tendency and culture to create small libraries that are lightweight abstractions over simple types.
Contrast this to OO languages where codebases tend to evolve massive interdependent interfaces of complex types, where the arguments and return types are more complex types and the complexity is fractal, each type is a class defined in terms of methods with arguments and return types or more complex return types.
It's not that OO type systems force fractal complexity to happen, but they encourage it, they make it easier to do the wrong thing. C doesn't make it impossible, but it makes it harder. C tends to breed simpler, shallower types with fewer dependencies that are easier to understand and debug.
Speed King
C is the fastest language out there, both in micro and in full stack benchmarks. And it isn't just the fastest in runtime, it's also consistently the most efficient for memory consumption and startup time. And when you need to make a tradeoff between space and time, C doesn't hide the details from you, it's easy to reason about both.

"Trying to outsmart a compiler defeats much of the purpose of using one."
- Kernighan & Plauger, The Elements of Programming Style

Every time there is a claim of "near C" performance from a higher level language like Java or Haskell, it becomes a sick joke when you see the details. They have to do awkward backflips of syntax, use special knowledge of "smart" compilers and VM internals to get that performance, to the point that the simple expressive nature of the language is lost to strange optimizations that are version specific, and usually only stand up in micro-benchmarks.
When you write something to be fast in C, you know why it's fast, and it doesn't degrade significantly with different compilers or environments the way different VMs will, the way GC settings can radically affect performance and pauses, or the way interaction of one piece of code in an application will totally change the garbage collection profile for the rest.
The route to optimization in C is direct and simple, and when it's not, there are a host of profiler tools to help you understand why without having to understand the guts of a VM or the "sufficiently smart compiler". When using profilers for CPU, memory and IO, C is best at not obscuring what is really happening. The benchmarks, both micro and full stack, consistently prove C is still the king.
Faster Build-Run-Debug Cycles
Critically important to developer efficiency and productivity is the "build, run, debug" cycle. The faster the cycle is, the more interactive development is, and the more you stay in the state of flow and on task. C has the fastest development interactivity of any mainstream statically typed language.

"Optimism is an occupational hazard of programming; feedback is the treatment."
- Kent Beck

Because the build, run, debug cycle is not a core feature of a language, it's more about the tooling around it, this cycle is something that tends to be overlooked. It's hard to overstate the importance of the cycle for productivity. Sadly it's something that gets left out of most programming language discussions, where the focus tends to be only on lines of code and source writability/readability. The reality is the tooling and interactivity cycle of C is the fastest of any comparable language.
Ubiquitous Debuggers and Useful Crash Dumps
For pretty much any system you'd ever want to port to, there are readily available C debuggers and crash dump tools. These are invaluable to quickly finding the source of problems. And yes, there will be problems.

"Error, no keyboard -- press F1 to continue."

With any other language there might not be a usable debugger available and less likely a useful crash dump tool, and there is a really good chance for any heavy lifting you are interfacing with C code anyway. Now you have to debug the interface between the other language and the C code, and you often lose a ton of context, making it a cumbersome, error prone process, and often completely useless in practice.
With pure C code, you can see call stacks, variables, arguments, thread locals, globals, basically everything in memory. This is ridiculously helpful especially when you have something that went wrong days into a long running server process and isn't otherwise reproducible. If you lose this context in a higher level language, prepare for much pain.
Callable from Anywhere
C has a standardized application binary interface (ABI) that is supported by every OS, language and platform in existence. And it requires no runtime or other inherent overhead. This means the code you write in C isn't just valuable to callers from C code, but to every conceivable library, language and environment in existence.

"Portability is a result of few concepts and complete definition"
- J. Palme

You can use C code in standalone executables, scripting languages, kernel code, embedded code, as a DLL, even callable from SQL. It's the Lingua Franca of systems programming and pluggable libraries. If you want to write something once and have it usable from the most environments and use cases possible, C is the only sane choice.
Yes. It has Flaws
There are many "flaws" in C. It has no bounds checking, it's easy to corrupt anything in memory, there are dangling pointers and memory/resource leaks, bolted-on support for concurrency, no modules, no namespaces. Error handling can be painfully cumbersome and verbose. It's easy to make a whole class of errors where the call stack is smashed and hostile inputs take over your process. Closures? HA!

"When all else fails, read the instructions."
- L. Lasellio

Its flaws are very very well known, and this is a virtue. All languages and implementations have gotchas and hangups. C is just far more upfront about it. And there are a ton of static and runtime tools to help you deal with the most common and dangerous mistakes. That some of the most heavily used and reliable software in the world is built on C is proof that the flaws are overblown, and easy to detect and fix.
At Couchbase we recently spent easily 2+ man/months dealing with a crash in the Erlang VM. We wasted a ton of time tracking down something that was in the core Erlang implementation, never sure what was happening or why, thinking perhaps the flaw was something in our own plug-in C code, hoping it was something we could find and fix. It wasn't, it was a race condition bug in core Erlang. We only found the problem via code inspection of Erlang. This is a fundamental problem in any language that abstracts away too much of the computer.
Initially for performance reasons, we started increasingly rewriting more of the Couchbase code in C, and choosing it as the first option for more new features. But amazingly it's proven much more predictable when we'll hit issues and how to debug and fix them. In the long run, it's more productive.
I always have it in the back of my head that I want to make a slightly better C. Just to clean up some of the rough edges and fix some of the more egregious problems. But getting everything to fit, top to bottom, syntax, semantics, tooling, etc., might not be possible or even worth the effort. As it stands today, C is unreasonably effective, and I don't see that changing any time soon.
Follow me on Twitter for more of my coding opinions and updates on Couchbase progress.

Posted January 8, 2013 1:00 PM