knowledge-kitchen

Software Engineeering - Historic Overview From Alan Turing to Today's Industrial Code Fry Cooks

What is it? Why is it useful?

  1. Attempts at a definition
  2. Challenges
  3. The Software Crisis
  4. Intrusion of DevOps
  5. Development lifecycle
  6. Process models
  7. Conclusions

Attempts at a definition

Wikipedia

Software engineering is the application of engineering to the development of software in a systematic method.

— Wikipedia

IEEE

The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software.

— IEEE Standard 610.12

ACM

Software engineering is concerned with developing and maintaining software systems that behave reliably and efficiently, are affordable to develop and maintain, and satisfy all the requirements that customers have defined for them.

— ACM

Merriam-Webster

A branch of computer science that deals with the design, implementation, and maintenance of complex computer programs.

— Merriam-Webster

Foo Barstein

Software engineerings is the systematic approach to creating software that uses empirical, evidence-based social and practical knowledge to invent, innovate, design, build, maintain, research, and improve software.

— Foo Barstein

Confusion by job title

Job titles vary wildly in industry. For example, each of these titles may or may not be for the role and responsibilities of a software engineeer - there is no way to know.

Definition by antithesis: bespoke development

Today, the antagonist to our protagonist is the bespoke developer - an individual or team who work take a unique custom-tailored approach to every project.

Definition by antithesis: bespoke development

It’s tempting to see the archetypal Agile developer as a revival of the long-haired countercultural weirdo who lurked around the punch card machines of the late 1960s. But the two personas differ in important respects. The eccentrics of computing’s early years wanted to program for the sheer thrill of putting this new technology to work. The coder of Agile’s imagination is committed, above all, to the project. He hates administrative intrusion because it gets in the way of his greatest aspiration, which is to do his job at the highest level of professional performance.

Definition by antithesis: bespoke development

The ideal of artisanal work is much pined for for its uniqueness and attention to detail.

But artisanal approaches do not scale well because…

Challenges

Software is abstract

Linus Torvalds (creator of Git and the Linux kernel), CNN interview from 2000:

You have much more freedom than you have when you’re building a bridge or a building. You can pretty much make up your own rules. […] It is artistry… at least for good programmers.

The bits and bytes on the machine are not constrained by physical limitations, limits in manufacturing, or transportation logistics. This allows for the development of a wild variety of systems - virtually any idea can be turned into software with little overhead.

This same lack of constraints allows for glutonous programmers to write code that can become increasingly complicated, deep, and difficult to maintain.

Software is increasingly complex

Fred Brooks, from “The Mythical Man-Month”:

Adding manpower to a late software project makes it later.

Creating quality software is hard, especially on a large scale.

Software is not just a program

Well-designed, functional, maintainable software usually includes many artifacts besides a program:

Demand for software is increasing

From the Bureau of Labor Statistics:

Demand for software is increasing (continued)

From the Bureau of Labor Statistics - the same in both 2018 and 2020:

Software must ideally be delivered on-time on-budget

Engineering is the application of science and knowledge to practical applications.

Software engineering aims to produce real-world software that can be used by its customers to solve their problems.

There is almost universally a budget and a deadline. Engineers must figure out how to solve the problem with those constraints.

Crises in software development

Alan Turing

In 1947, in a lecture to the London Mathematical Society, computing pioneer, Alan Turing, said,

One of our difficulties will be the maintenance of an appropriate discipline, so that we do not lose track of what we are doing. We shall need a number of efficient librarian types to keep us in order…

-Alan Turing

Alan Turing is considered the inventor of the computer program as we know it today. In 1947, he could have been considered the only computer programmer in the world.

Alan Turing

Another prescient quote from Turing’s lecture:

The masters [programmers] are liable to get replaced because as soon as any technique becomes at all stereotyped it becomes possible to devise a [program] which will enable the electronic computer to do it for itself. It may happen however that the masters will refuse to do this. They may be unwilling to let their jobs be stolen from them in this way. In that case they would surround the whole of their work with mystery and make excuses, couched in well chosen gibberish, whenever any dangerous suggestions were made.

-Alan Turing

The Software Crisis

Turing’s premonition of software engineering as a discipline came of age during the turbulent times of the 1960’s, as more and more people began to recognize it was a rare project which resulted in efficient and well-functioning software in a reasonable amount of time at a reasonable cost.

The Software Crisis

“Software’s Chronic Crisis”, Scientific American, Sept. 1994:

By the time [the art of programming] reached 25, the difficulties of building big software loomed so large that in the autumn of 1968 the NATO Science Committee convened some 50 top programmers, computer scientists and captains of industry to plot a course out of what had come to be known as the software crisis. Although the experts could not contrive a road map to guide the industry toward firmer ground, they did coin a name for that distant goal: software engineering, now defined formally as “the application of a systematic, disciplined, quantifiable approach to the development, operation and maintenance of software.”

A quarter of a century later software engineering remains a term of aspiration.

C.A.R. Hoare

In 1980, on his acceptance of the Turing Award for “fundamental contributions to the definition and design of programming languages”, C.A.R. Hoare said,

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.”

-C.A.R. Hoare

Intrusion of DevOps

Merging Development with Operations

DevOps is the amalgamation of…

Formerly considered disparate disciplines, in contemporary thought, operations functions are either incorporated into the workload of developers, or managed by IT specialists working in close communication with developers.

Relying on automation

How Operations can be merged with Development is largely explained by the rise of automation.

Automated tools and virtualization now allow developers to trigger formerly time- and resource-consuming IT operations with minimal setup and wait time and minimal reliance on humans.

Relying on automation

Operations such as the following are now often done with the assistance of automated tools:

Development lifecycle

Development lifecycle

Phases of development

Regardless of process model, the phases of development are familiar to all:

  1. Specification
  2. Implementation
  3. Validation
  4. Evolution

Development lifecycle

Specification

Figuring out as much as you can about what the system will do and how it will do it.

Development lifecycle

Implementation

Writing the code, designing the interface, writing the text, building the systems, making it work according to plan, etc.

Development lifecycle

Validation

Does it work? Is it what you planned?

Development lifecycle

Evolution

Process models

Overview

Big Design Up-Front

In Big Design Up-Front methodologies, such as “Waterfall”, we perform each of the lifecycle phases once for the entire project at inception.

Waterfall process lifecycles

Big Design Up-Front

With Big Design Up-Front/Waterfall, once a specification has been drafted and agreed upon (‘signed-off on’), estimating project cost and completion date is, in theory, straightfoward, since everyone has agreed exactly what is to be done and how it will be done.

Big Design Up-Front

In Big Design Up-Front/Waterfall methodologies, each phase must be completed and ‘signed off’ before the next phase can begin.

Since there is only one iteration of each phases, these phases typically take a long time. The project needs might change during that time.

Any change requires a rewriting of the entire specification and another ‘sign-off’ from the client and re-doing of any work already done in subsequent phases.

Incremental methodologies

In Incremental development methodologies, such as the so-called “[Agile](/content/courses/agile-development-and-devops/slides/scrum/#agile-manifesto” methodologies, the work to be done is broken up into increments, and developers repeat these phases many times for each piece/increment of the project at discrete intervals.

Iterative process lifecycles

Incremental methodologies

In Incremental/Agile methodologies, since work is broken up into discrete increments with their own lifecycle phases, any changes to the project only affect the current iteration, not the project as a whole.

Incremental methodologies

In Incremental/Agile methodologies, since specification and the other phases only happen in increments, there is no holistic view of how long the project will take nor how much it will cost.

Incremental/Agile development, in a purist sense, therefore is only feasible when unlimited money and unlimited time are available.

Incremental methodologies

Since unlimited funds and time are usually not available, many Incremental/Agile development practitioners put fixed time limits and money limits on the project, and try to complete as much as possible in that time for that money.

Incremental methodologies

Since the project may not be fully completed before time and money run out, Incremental/Agile developers try to have each increment be a runnable product that may not be feature-complete, but will be functional as-is, even if no more work is to be done on the project.

Conclusions

Thank you. Bye.