The most important factor to make a software project success

We often talk about resource, budget, timing, quality when talking about project management. I treat project scope management as the most important factor among all the project management factors, more important than factors like schedule, cost, budget, …Since I saw too many times what project team delivered is not what customer wants.

 

Project scope management is essentially acting as the bridge between the project team and outside world, especially customer. The project manager has the responsibility to make sure the project team is doing something expected by business team, or customer. No budget, resource wasted on activities not align with customer request. There are maybe some project sponsored by internal authorities, but still there should be scope statement from the internal authority.

 

The project manager has the responsibility to communicate/control the sub-module scope among the project team.  There are 3 good practices for doing this:

1. Document the scope in structured, better itemized statement list;
2. Organize all stake holders to present the scope in understandable chart;
3. Publish the scope statements and charts to a place easily accessible and editable by relevant stakeholders.

In summary, project manager should ensure project team members are working towards the correct scope at any project stage. I think anyone has experience on software development team with more than 3 software Engineers knows the difficulty to actually achieve this goal.

Project Charter Example

I always wonder how to specify project charter clearly and efficiently (no excessive document), until I see the example below.

1. Project Title and Description

What is the project?

Customer Satisfaction Fix-It Project: Over the last few months the quality assurance department has discovered many of our customers' orders for our ABC equipment have taken the customer ten times longer to place through our computer network than our competitors' networks. The purpose of this project is to investigate the reasons for the problem and propose a solution. The solution will be authorized as a subsequent project. Quality Control has detailed records of their findings that can be used to speed up this project.

2. Project Manager Assigned and Authority Level

Who is given authority to lead the project, and can he/she determine, manage and approve changes to budget, schedule, staffing, etc.?

Alexis Sherman shall be the project manager for this project and have authority to select team members and determine the final project budget.

3. Business Need

Why is the project being done?

This project is being completed in order to prevent a further breakdown of customer satisfaction.

4. Project Justification

Business case-On what financial or other basis can we justify doing this project?

We expect that improved customer satisfaction will increase revenue to the company in the first year by at least $200,000 due to a decrease in service calls. As a side benefit, we hope that the project will generate ideas on improving customer
satisfaction while fixing this problem.

5. Resource Pre-assigned

How many or what resources will be provided?

Morgan and Danny are already dedicated to the project because of their expertise in computer networks of this type. Other resources will be determined by the project manager.

6. Stake-Holders

Who will affect, or be affected by the project (influence the project), as known to date?

Stakeholders include Connor representing Quality Control, Ruth in Customer Service and Mary in Marketing. These resources are available to assist the project as needed by the project manager.

7. Stake-holder Requirements as Known

Requirements related to both project and product scope.

Attached to this document are the detailed specifications for the existing system, the requirements that the existing system was designed to meet. It is expected that this project will not change how the system affects the existing requirements. The project must include utilizing the data available from quality control.

5. Product Description/Deliverables

What specific product deliverables are wanted and what will be the end result of the project.

1. A report that outlines what can be changed, how much each change will cost and expected decrease in the time it takes to place and order resulting from each change. Few words are necessary in the report, but it must be created electronically and be agreed to by the heads of quality control, customer service and marketing in addition to the project team.
2. A list of the interactions with our customers necessary to complete the changes. A work breakdown structure, due within two weeks, that outlines the plan for accomplishing the project, followed one week later by a list of risks in completing the project.

6. Constraints and Assumptions

A constraint is any limiting factor and an assumption is something taken to be true, but which may not be true.

Complete the project no later than <date>. Spend no more than <certain amount money>. We have assumed that Kerry will be available to assist the project and that testing can be done on the seller’s computer.

Project Sponsor Approval:

<Signature>

Find and sort acronyms for Microsoft Word document automatically.

We guys work on Information Technology. I always have too many acronyms to deal with when I create a document.  There are many solutions of adding acronym to a word document, such as add as a Bookmark or Hyperlink. But most of them require the author to go through the document and mark all the acronyms manually, which I don’t like.

This script searches through your word document and find all Words >= 3 (configurable) alphabets in Uppercase, then remove duplicate entries and sort by alphabetical order.  This is exactly what I need.  If  you want to try, click Tools->macros->Visual Basic Editor, copy-paste the code below and execute it. There you go!

 

Sub ExtractAcronymsToANewDocument()

Dim oDoc_Source As Document
Dim oDoc_Target As Document
Dim strListSep  As String
Dim strAcronym  As String
Dim oTable      As Table
Dim oRange      As Range

Dim n As Long
Dim strAllFound As String
'use to keep track of founded. Find the list separator from international settings
'In some countries it is comma, in other semicolon

strListSep = Application.International(wdListSeparator)
strAllFound = "#"

Set oDoc_Source = ActiveDocument
'Create new document for acronyms
Set oDoc_Target = Documents.Add

With oDoc_Target
'Make sure document is empty
.Range = ""
'Insert a table with room for acronym and definition
Set oTable = .Tables.Add(Range:=.Range, NumRows:=2, NumColumns:=3)
    With oTable
    'Format the table a bit
    'Insert headings
    .Cell(1, 1).Range.Text = "Acronym"
    .Cell(1, 2).Range.Text = "Definition"
    .Cell(1, 3).Range.Text = "Page"
    'Set row as heading row
    .Rows(1).HeadingFormat = True
    .Rows(1).Range.Font.Bold = True
    .PreferredWidthType = wdPreferredWidthPercent
    .Columns(1).PreferredWidth = 20
    .Columns(2).PreferredWidth = 70
    .Columns(3).PreferredWidth = 10
    End With
End With

With oDoc_Source
Set oRange = .Range
n = 1 'used to count below

    With oRange.Find
    .Text = "<[A-Z]{3" & strListSep & "}>"
    .Forward = True
    .Wrap = wdFindStop
    .Format = False
    .MatchCase = True
    .MatchWildcards = True
    Do While .Execute
    'Continue while found
    strAcronym = oRange
    'Insert in target doc
    'If strAcronym is already in strAllFound, do not add again
    If InStr(1, strAllFound, "#" & strAcronym & "#") = 0 Then
        'Add new row in table from second acronym
        If n > 1 Then oTable.Rows.Add
        'Was not found before
        strAllFound = strAllFound & strAcronym & "#"
        'Insert in column 1 in oTable
        'Compensate for heading row
        With oTable
        .Cell(n + 1, 1).Range.Text = strAcronym
        'Insert page number in column 3
        .Cell(n + 1, 3).Range.Text = oRange.Information(wdActiveEndPageNumber)
        End With
        n = n + 1
        End If
        'If acronym
    Loop
    End With
End With

'Sort the acronyms alphabetically
With Selection
    .Sort ExcludeHeader:=True, FieldNumber:="Column 1", SortFieldType _
    :=wdSortFieldAlphanumeric, SortOrder:=wdSortOrderAscending
    .HomeKey (wdStory)
End With

'Clean up
Set oDoc_Source = Nothing
Set oDoc_Target = Nothing
Set oTable = Nothing
MsgBox "Finished extracting " & n - 1 & " acronymn(s) to a new document."

End Sub

Breakdown of Initiating-Planning-Executing-Controlling-Closing process

I got this list of possible project tasks.  personally I think it is a good practice to go through this list, and create marks on the project current status along the way.

Initiating

• Select Project Manager.
• Determine company culture and existing systems.
• Collect processes, procedures and historical information.
• Divide large projects into phases.
• Identify stake-holders.
• Document business need.
• Determine project objectives.
• Document assumptions and constraints.
• Develop project charter.
• Develop preliminary project scope statement.

I am working in technical domain, or more specifically,  embedded system development. I think it is vital important to verify architecture correctness while creating the project charter also. The best way is to do some prototype work, though maybe too luxury. To the bear minimum, the project manager should create some diagram and discuss with relevant stakeholders,  otherwise is just waste the organizations and money and time, and market opportunity.

 

Planning

• Determine how you will do planning – part of management plans.
• Create project scope statement.
• Determine team.
• Create WBS and WBS dictionary.
• Create activity list.
• Create network diagram.
• Estimate resource requirements.
• Estimate time and cost.
• Determine critical path.
• Develop Schedule.
• Developer budget.
• Determine quality standards, processes and metrics.
• Determine roles and responsibilities.
• Determine communications requirements.
• Risk identification, qualitative and quantitative risk analysis and response planning.
• (items above this lime will need Iterations ).
• Determine what to purchase.
• Prepare procurement documents.
• Finalize the “how to execute and control” aspects of all management plans.
• Create process improvement plan.
• Develop final PM plan and performance measurement baselines.
• Gain formal approval.
• Hold kick-off meeting.

My personal feeling is better to use 1/10 criteria for planning, that is, to estimate at the granularity level of 1/10. For example,  setup a monthly schedule for a half year to one year project, or create biweekly schedule for a one quarter project, or create a WBS with 10 to 20 tasks. I have seen task break down with several hundred tasks, eventually it becomes a game to figure out which activity goes to which task, and really a brain test for entire team.

Executing

• Acquire final team.
• Execute the PM plan.
• Complete product scope.
• Recommend changes and corrective actions.
• Send and receive information.
• Implement approved changes, defect repair, preventive and corrective actions.
• Continuous improvement.
• Follow processes.
• Team building.
• Give recognition and rewards.
• Hold progress meetings.
• Use work authorization system.
• Request seller responses.
• Select Sellers.

In my opinion, there are two key points for executing a project: 1. People management: understand what your colleagues can do and try to balance the workload. I think making presentation and/or have frequent chit-chat are good ways to bring the gap. At least I feel more clear of one concept after I can make others understand what I am talking about.  2. Build an adaptive process: Nobody can plan everything accurately at the beginning. I like the practice to check the status at the end of each week (or biweekly), and figure out how to make up next week. I think it is called iteration process or spiral model officially.

Monitoring & Controlling

• Measure against the performance measurement baselines.
• Measure according to the management plans.
• Determine variances and if they warrant corrective action or a change.
• Scope verification.
• Configuration management.
• Recommend changes, defect repair, preventive and corrective actions.
• Integrated change control.
• Approve changes, defect repair, preventive and corrective actions.
• Risk audits.
• Manage reserve.
• Use issue logs.
• Facilitate conflict resolution.
• Measure team member performance.
• Report on performance.
• Create forecasts.
• Administer contracts.

For Software/Firmware Development, There are organizations that project manager is busy working on collecting all the matrix, quality team building nice diagram/chart subsequently, and top management evaluating the organization / individual performance based on these charts. In the end, Engineers have to cook data so that the final evaluation will be good for them. I believe the role for project manager is to ensure the accuracy of monitoring. BTW: I also think cooking data is hard for Small and Medium Organization because SMEs always face customer directly.

Closing

• Develop closure procedures.
• Complete contract closure.
• Confirm work is done to requirement.
• Gain formal acceptance of the product.
• Final performance reporting.
• Index and archive records.
• Update, lessons learned and knowledge base.
• Hand off completed product.
• Release resources.

Not many project managers are lucky enough to lead the project until its official ending. However, I think a responsible project manager for an organization should collect historical information for this project, and put into a way easy to be utilized by the next project manager. Especially for a software development organization, attributes like Engineer productivity, defect density, percentage of each stage, and domain expertise are important for the sustainability of the organization.

Overlap of Processes along the project timeline

This diagram depicts the relationship of processes well, I like it.

Projectized versus functional organization

An organization can be either functional oriented or project oriented.

Organization Structure	Projectized	Matrix	Functional
Domain Expertise	Poor	Medium	Good
Project Coordination	Good	Medium	Poor

In practical, most of organizations are matrix based, i.e, team is organized by functional area, and team members work for projects most of the time under project manager.

The pros and cons of a project organization is listed below, although the truth is matrix organization can be a nightmare for Engineers.

Pros	Cons
Highly visible project objectives	Extra administration required
Improved project manager control over resources	More than one boss for project teams
More support from functional organizations	More complex to monitor and control
Better horizontal and vertical dissemination of information than functional	Functional managers may have different priorities than project managers
Team members maintain a “home”	Higher potential for conflict
Better coordination	Need extensive policies and procedures
Maximum utilization of scarce resources	Tougher problems with resource allocation

The life cycle of Process, Project, Program, Product

Process: a series of actions bringing about a result. Process is repeatable within an organization.

Project: a project is a temporary endeavor undertaken to create a unique product or service.  The project life cycle refers to a logical sequence of activities to accomplish the projects goals or objectives. 

Program: a program is a group of projects managed in a coordinated way to obtain benefits not available from managing them individually. Sometimes a program management and a project management can be treated as synonyms. A program management can also be treated as superset of subset of project management depend on situations. 

Product: The life cycle lasts from the conception of a new product to its withdrawal.

Light-weight CMMi deployment

I was talking to one of my ex-supervisor in Motorola on his experience of deploying CMMi to another organization out of Motorola. There are some interesting findings about customizing CMMi for different organization cultures.  As Motorola has officially announced closing down of its Singapore Software Center, now we can comment a bit on their CMMi process, and what other organizations can learn from it.

Motorola is famous for the CMMI level-5 deployment to its various software centers across the world. These software centers are such process oriented organizations that low to mid management level take it for granted that the software requirement, design, implementation and testing document must be in place, and process audit must be in place. There are plenty of diagrams, analysis reports generated that becomes over-complicated, for good and bad.

In early 1990s, Software was such a special technical skill that only a small group of elite people can do it well, so Motorola setup various software centers around the world, with software process deployment initiative. Both the organization model and the process proved to be quite effective initially. Later as the software complexity grows, software process also grows to ensure the quality and efficiency of the code, which is excellent. However, these centers are far away from the end customers. The marketing team cannot transfer their pressure to software developers effectively in this organization structure.  The mid-level management turns to process focus, instead of customer focus, because that is the instruction implied by the evaluation process (for sure top management still emphasize they are marketing oriented).  All software projects are internal, which are quite easy going. When the profit margin of software business goes down and the company eco-environment got worse, eventually the company cannot afford this over-complicated business model and close down the center. 

This does not mean it is not possible to marry the good merit of process-oriented approach to innovation-oriented approach. One compromise way we found is to simply the CMMi process and deploy it implicitly. More specifically, requirement and software configuration management are the two most important KPAs a software organization needs to pay attention to, next comes implementation and test define. i.e, to implement what you are required to implement, and test what you are required to test. By telling your customer about implement customer requirement, and control your software team so that the entire team works towards the same goal, instead of working against each other (not personally, but because either Architect or process has flaw, I have seen a lot examples that people work against each other.),  you have the CMMI corner stone lay out.

The process effort should not exceed 5% of the total project effort. If there is no full time process team, then do 2 things:

1. have one internet-savvy young engineer to update and maintain an intranet to publish requirements (it is important to put requirements, designs, … into a common repository). 

2. List the key process flow in one Page (A4), make sure the font size is readable, let the team has free access to it, and have an experienced Engineer to audit regularly (biweekly or monthly).  A white paper (such as Software Development Manual) more than 2 pages is only useful for process team, nobody in the project team will read it. So DON’T waste your time.

In summary, the ultimate goal for an organization is not to reach certain CMMi level (though it is a side output, necessary for many human performance evaluation), but to make the final output predictable and quantifiable. Process is like lubricating oil, you won’t feel it if your machine running smoothly, and it is actually performing its duty implicitly. The most important thing: Software Process must come from real-life experience.  There are endless documents out there, people won’t convinced if you are only repeating a book story.

Common Errors for a project manager

If you have ever done be a part of a project team, do you experience any kind of scenarios below? If so, I am with you. However, I do not think these are all project manager’s fault.

1. Focusing on asking for percent complete
2. Holding "go around the room" type status meetings
3. Spending host of your time babysitting team members by constantly checking on them
4. Asking to cut lo percent off the estimate
5. Not attempting to obtain finalized requirements
6. Not getting real resource commitments
7. Not having a reward system
8. Not focusing on quality
9. Not having a control system
10. Not having management plans
11. Not measuring against the project management plan, or even creating metrics
12. Not spending time finding and eliminating root causes of problems or deviations
13. Not implementing corrective action to keep the project in line with the project management plan
14. Not reevaluating the effectiveness of the project management plan
15. Not reevaluating the accuracy or completeness of schedule, cost, scope
    Ignoring resource managers' need to have their people do their own departments' work
16. Not realizing the project can affect the reputation of team members
17. Not realizing the project manager has some human resource responsibilities to the project team, such as project job descriptions and adding letters of recommendation to team members' human resource files
18. Blaming unrealistic schedules on management instead of realizing they are the project manager's responsibility

My understanding of Project Management -ism

1. Historical records are exceedingly valuable (like gold) to the project manager, the team, the performing organization and even the customer.
2. Project cost and schedule cannot be finalized without completing risk management.
3. A project manager must work within the existing systems and
   culture of a company, which is referred as enterprise environmental factors and they are inputs to many processes.
4. A project manager spends all his time dealing with problems is not a great project manager. A good project manager plans the project to address the problems and to prevent the problems coming.
5. Percent complete is an almost meaningless number. [JS] Unfortunately there are many cases that upper management only wants a number.
6. A great project manager does not hold meetings where you go around the room asking all attendees to report. Such meetings are generally, but not always, a waste of time. [JS] It is a channel for team members to clarify their concerns of the project.
7. The project management plan is approved by all parties, is realistic and everyone believes it can be achieved. [JS] I think it is a good practice to make the project management plan, or at least store it at a shared place.
8. If at all possible, all the work/assignment and all the stakeholders are identified before the project begins. Stakeholders are involved in the project and may help identify and manage risks. [JS] A project success is all project team member’s responsibility, not only the project manager.
9. The project manager has some human resource responsibilities of which you might not be aware. [JS] Whether you have the right person and they are keen to the project, and how team members are rewarded for their work. However, a project management is always limited by the organization.

Comments on “Things every project manager should do”

I read through a statement that “you do not understand project management if you do not understand five or more of the following items”. The list makes great sense to me, so I list here and add my own comments in italic.

1. A step-by-step process for managing projects and why each step is necessary.   I will only be confident in iterative model.
2. Project manager, sponsor and team roles.
3. Historical information from previous projects.
4. Lessons learned from previous projects.
5. Creation of lessons learned on your projects.
6. Project charter. Everyone must understand the same charter and work towards the same goal. This is not so strait forward during the project execution.
7. What is a work breakdown structure, how to create it, and that it is not a list in a bar chart. My understanding is this means the breakdown is not a list on paper, the team member should understand the meaning.  A good practice is to break down into 5 to 8 sub tasks in terms of time and size. For example, a quarter duration should be break down into biweekly tasks.
8. How to manually create a network diagram Don’t understand this.  How it is related to project management. Does it talking about human network.
9. Critical path-how to find it and what benefits it provides the project manager. Key points
10. Three-point estimating: To estimate a value (cost, duration, etc.), assume a=the best-case estimate m=the most likely estimate b=the worst case estimate, then the weighted average E=(a+4m+b)/6, and standard deviation=(b-a)/6
11. Monte Carlo analysis： refers specifically to a technique in which the project team leader or project team computes and/or quantifies the complete and total project cost and/or project schedule a number of times through the use of input values that have been selected at random through the careful utilization of probability distributions or potential costs and/or potential durations. I think it is an iterative way to define project cost or schedule.
12. Earned value. refers to the actual methodology of management in which the project management team embarks in the process of integrating the scope, the schedule, and the resources that are determined to be needed in the process of making an objective measurement of the progress that has taken place to date on a project, and also on how successful the performance has been to date. Performance is measured by assessing the budgeted cost of all work that has been performed to date (also known as earned value, or EV) and comparing it to the actual cost of work that had been performed to date (which is also known as the actual cost). Progress is determined by comparing the earned value to date and measuring it against the planned or expected value. Earned value management is essential to maintaining a proper budget.
13. Schedule compression, crashing and fast tracking
14. An unrealistic schedule is the project manager's fault. Mostly because the project scope, dependency, team capability not well understood.
15. Creating a realistic and approved project management plan that you would be willing to be held accountable to achieving.  At least for firmware project, a realistic plan is only possible if the domain and scope are understood. I don’t dare to say fully here, as in many cases, it is poorly understood.
16. Measuring and implementing corrective action.
17. Risk management process and that risk management is not just using a checklist.
18. Expected monetary value. Agree
19. Calculating budget reserves and their relationship to risk management.
20. Controlling the project to the project management plan. 

Do you know enough about project management?

I am reading PMP Preparation book recently. It has one statement:  You do not know enough if you experience two or more of the following problems on projects: .

1. Large cost or schedule overruns ;
2. Unrealistic schedules;
3. Excessive changes to the scope or schedule;
4. Poor communications and increased conflict;
5. Running out of time near the end of the project;
6. Unsatisfactory quality;
7. Low morale;
8. People on the team are unsure of what needs to be done;
9. Excessive rework and overtime;
10. Too many project meetings;

I totally agree that these are problems for project management. However, I do not think the project team can totally eliminate these problems even the project manager and team members have excellent technical and process knowledge of the project.

A project team is not isolated in vacuum environment. The project execution is restricted by many other factors. For example:

1. A giant MNC may have corporate wide pay-cut and promotion freeze, which might cause low morale of the project team, and the project manager can do nothing.
2. Excessive changes to the scope and schedule maybe because sales team boast some excessive functionality, which cannot be handled by Engineering team in time.
3. Too many project meetings maybe because the organization has a complex matrix structure, which is beyond the control of this team.

Actually for each items listed by PMP book above, there could be some reason beyond the project manager’s control. Anyway, I think what a project manager can do is to try his best to mitigate these problems, although it is really hard to eliminate them all.

Differences between Java and C++ in terms of OOP

Most  people start learning Object Oriented Programming from either Java and/or C++.  Although the concept of Object Oriented Programming is the same (in fact, you can even do Object oriented programming without use any OO language), there are subtle difference between Java and C++ from OOP perspective.  I try to list some of the key differences for reference, you are welcome to comment. Please note the grammar differences are not the interest point of this blog.

All stand-alone C++ programs require a function named main and can have numerous other functions. Java does not have stand alone functions, all functions (called methods) are members of a class. All classes in Java ultimately inherit from the Object class, while it is possible to create inheritance trees that are completely unrelated to one another in C++.  In summary, Java is a pure Object oriented language, while C++ is a mixture of Object oriented and structure language.

The interface keyword in Java is used to create the equivalence of an abstract base class containing only method declarations and constants. No variable data members or method definitions are allowed. C++ does not support interface concept. Java does not support multiple inheritance. To some extent, the interface feature provides the desirable features of multiple inheritance to a Java program without some of the underlying problems.

Java is running on a Virtual Machine, which can recollect unused memory to the operating system, so Java does not destructor.  Unlike C++, Java cannot access pointers to do memory operation directly. This leads to a whole host of subtle and extremely important differences between Java and C++. 

Furthermore, the C++ compiler does not check whether all local variables are initialized before they are read. It is quite easy to forget initializing a variable in C++. The value of the variable is then the random bit pattern that happened to be in the memory location that the local variable occupies.

Java does not have global functions and global data. Static in Java is just like global in C++, can be accessed through class name directly, and shared by all instances of the class.  For C++, static data members must be defined out side of class definition, because they don't belong to any specific instance of the class.

Generally Java is more robust than C++ because:

• Object handles (references) are automatically initialized to null.
• Handles are checked before accessing, and exceptions are thrown in the event of problems.
• You cannot access an array out of bounds.
• Memory leaks are prevented by automatic garbage collection.

While C++ programmer clearly has more flexibility to create high efficient program, also more chance to encounter error.

Difference between Stack memory and Heap Memory.

Recently I came across the difference between stack/heap topic. In simple, Both are dynamic memory allocated for program execution, but not the only 2 memory regions allocated for program execution.

Please Google “stack, heap” for the definition, summary for the difference:

Heap
- free-list - list of free space
- on allocation - memory manager finds space and marks it as used changing free-list
- on de-allocation - memory manager marks space as free changing free-list
- memory fragmentation - memory fragments into small blocks over lifetime of program
- garbage collection - coalesce fragments, possibly moving objects (must be careful of pointers when moving!)
- Concept at Operating system Memory management layer.

Stack
- clean and efficient support for nested functions and recursion
- central concept is stack frame (also called activation record)
- Concept at Micro Processor hardware registers layer.

Simple Example:

void foo()
{
    int x;                     <<< x is on the stack
    char *ptr = new char[255]; <<< 255 characters are allocated in the heap, but ptr object is on the stack
    int array[255];            <<< 255 ints are all on the stack
}

Travel

Please refer to my Chinese blog in China. This topic is safe to be put inside China.

Storage Area Network

1. What is a storage area network?

A storage area network (SAN) is an architecture to attach remote computer storage devices such as disk arrays, tape libraries to servers, in such as way that the storage devices appears locally to the operating system.

By contrast to a SAN, network-attached storage uses file-based protocols such as NFS or SMB/CIFS where it is clear that the storage is remote, and computers request a portion of an abstract file rather than a disk block.

A computer cluster is a group of tightly coupled computers that work together so that in many respects they can be viewed as through they are a single computer.  

2. Why we need storage area network?

....

3. How does a storage area network work?

Most storage networks use the SCSI protocol for communication between servers and disk drive devices, though they do not use its low-level physical interface, instead using a mapping layer, such as the FCP mapping standard.

• Fibre Channel, currently the most common. Comes in 1 Gbit/s, 2 Gbit/s, 4 Gbit/s, and 8 Gbit/s variants (10 Gbit/s is under development)
• iSCSI, mapping SCSI over TCP/IP
• HyperSCSI, mapping SCSI over Ethernet
• ATA over Ethernet, mapping ATA over Ethernet
• InfiniBand (IB), supports mapping SCSI over IB and/or mapping TCP/IP over IB

Programming

Most people think that programming is only writing several lines of code. Actually a good programmer needs not only to understand programming language syntax very well, but also to play around with many tools and build up good programming practice. There ARE some super star programmers, but super star programmers could spoil your delivery since only he knows everything. When it comes to the customer support stage, he will be too busy to answer all the questions. One day he left the company, Wow, nobody can take over his job. The good software must be the collaboration result of the development team, test team and many support teams, plus your customer.

I tried to express my software engineering experience from technical perspective, there is an article on software engineering from project organization perspective in this website also.

First of all, some software processing people believe that the “bug density” of software is from software processing and quality control only, it should be independent of people. Yes, it IS from process and quality. However, the project organization, tech leader’s technical capability and whether the team are willing to do the project (don’t laugh, it is true sometime people just get annoyed and don’t want to do anything).

1. GNU development tools on Linux. 1

2. Other Tools. 1

3. Programming Languages. 1

4. Good practices. 2

1. GNU development tools on Linux

GNU development tools are a complete set of framework. They are absolutely free and powerful. Some people say that there is nothing but the GNU way to develop software, a typical Linux/UNIX software developer uses in his everyday work. GNU development tools are not worse than any other commercial (expensive) so called "killer-apps" widely used on other platforms. GNU tools are your friends for any development work done on Linux. Before you want to purchase new software development tools, think it over, I bet you could have a GNU solution to satisfy your requirement.

GNU native tools

This set of tools are used for x86/Linux native development I used to play with.

GNU cross tools

Linux platform virtually supports everything required for embedded development. Some company had to use windows cross-development environment simply because historical reasons. You may find information on Linux cross development environment here.

2. Other Tools

Other tools

However, there are still some commercial tools you should know. Anyway, everyone want to use free tools and sell their product with high price. If nobody buy commercial tools, many software engineer will lost their job also.

3. Programming Languages

Programming Languages

4. Good practices

Good Practice

Embedded Linux Development Environment and System Programming

Purpose: This course is delivered to the second year Master students at the Software School of Beijing University  From July 15^th 2003 to Aug 28^th 2003

1. Objective

This course is designed for master students and embedded software engineers to have:

1. A preliminary understanding of key techniques and tools used to implement embedded software;
2. Hands-on experience on GNU tools used to develop embedded Linux based software;
3. Hands-on experience on basic features of embedded Linux system programming;

This course is not designed for the students to understand everything of embedded system, which is not possible also. Although we are only using GNU tools and powerpc platform as the example throughout the course, the students are expected to gain the capability to pickup other tools and target platform at short time frame based on their understanding of GNU environment and Powerpc platform.

This course is not intended to teach:

1. Software design process and methodology;
2. C/C++ syntax and Linux OS;
3. Target platform (Powerpc) architecture;

However, Understanding of these domains is pre-requirement for the course.

2. Course Outline

You can expect the following questions be answered during the course.

First of all, why we need the embedded system?

How special is the embedded system? Or How different it is from the desktop environment?

What is the development life-cycle of an embedded software project? Again, how it is different from a desktop software project?

What are the GNU tools used for a complete development life-cycle?

I have seen some tools not covered by this course, how could I understand these tools quickly?

What are the basic features to embedded Linux?

Why these features are important? What are the philosophyies behind?

My new embedded software project uses a scheduler I have never heard before, how do I analyze it?

3. Instruction Methods

This course is a hybrid of:

1. Lecturers;
2. On-class hands-on;
3. Group discussion;
4. Off-class projects.

4. References

How to make your WindowsOS execute faster?

My Windows XP becomes slower and slower over the time. Therefore, I had to google the internet and look for tools and tricks to optimize my Windows XP as much as possible. However, there fore many so called "Windows optimization tricks" does not work as it specified at the web-page. After several rounds of retry, I listed those steps I think useful for my Windows XP optimization.

1. Remove unnecessary software and components, especially Windows Index services;

    Go to Control Panel-> Add or Remove programs, check those rarely used software, whether it is installed intentionally or unintentionally, then uninstall those you don't need.

    Go to Control Panel-> Add or Remove programs -> Add/Remove Windows Components, remove those components you don't need. For example, Windows indexing service.

    Disable unnecessary start-up items. Google "Windows Startup Manager" will list tools to help on this topic.

2. Delete unnecessary files;

    Many Windows optimization tools offer some utility to clean up the unnecessary files. I am using this one: http://www.ccleaner.com/

    The tool can only delete temp files created by some installation process or Windows application. If you downloaded a file and later forgot it, the tool won't know it. So it is important for you to make your directory structure organized. For me, I create a <Root>/Data directory to store all human created data (versus directory created by installation binaries). I always download any article to <root>/Data/temp first, and only move valuable binary or documents to my permanent directory structure if I think it is useful.

3. Optimizing you Windows appearance

    Go to Control Panel->System->Advanced->performance Settings->Visual Effects, choose Custom and tick:

         Show shadows under menus

         Show translucent selection rectangle

         Smooth edges of screen fonts

         Use drop shadows for icon labels on the desktop

         Use visual styles on windows and buttons

    You may have some other preference, but I found this tick-list is most close to Windows default and what I feel comfortable.

4. Stop unnecessary Windows services:

    Windows starts a lot services by default, most of them are not necessary, so Go to Start->Cmd->services.msc and select Which services you want to stop. However,  many web page on optimization tricks ask you to go through the service one by one and stop a lot services, which maybe harder for novices.  Besides all the default setting, I only changed 3 of them to manual: Remote Registry, Telnet, Indexing Services.

5.  Clean up registry;

     It is dangerous to edit Windows registry (though sometimes I do), better find a tool to do it automatically.

6. Irq14=4096.

    It is said add Irq14=4096 will improve the hard-disk performance. I did so, but not sure whether my hard-disk performance is improved, nor did I figure out the reason. The paper said is increased the buffer for hard-disk read/write, but I did not figure out what is the default buffer setting.

7. De-fragment your hard-disk regularly;

    Find you C: or D: drive at Windows explorer, and click property, use your Windows tool to de-fragment the hard-drive, and reboot it. 

IT Services

Many people came to ask me about some IT configuration questions. This page lists most of my recommendation for configuring PC and SOHO network. It is open for discussion. Please send your comments to jinsheng_tang@yahoo.com

1. Setup your PC.

It is better to setup your PC dualboot with Windows XP and Linux if you are an IT professional or willing to be, otherwise please check the Windows section only.

It is easy to setup your PC with Windows and Linux dualboot, please google "config Windows Linux dualboot" for detailed procedures.

The recommendation here is:

1. Partition your hard disk with 8-10G for Windows OS, which will be used for install of Windows OS and application only.

You may wonder why so much disk space is required, it is reserved for multimedia tools, various Java virtual machines, documentation tools like MS office and Visual studio and Oracle tools.

You may need 2-4G more if you want to play around with Virtual PC, but it is not encouraged as I felt it is very slow and not worth for use. I'd rather use 2 PCs instead of setting up Virtual PC.

If you are not going to play around with Oracle or Java Virtual Machine or Visual Studio, I bet you need several multimedia tools. So reserve 5G at least.

2. Assign 6-8G for Linux

Redhat Linux 9.0 needs nearly 5.0G for a complete installation plus Linux swap and boot, You may want to deselect many language package during installation to save some disk space, or sort RPM by size (please google some perl script to do it) and remove unnecessary RPM with size over 1G. Eventually around 1G space could be saved.

I have encounter the problem that some automatic compile script or Makefile can not be executed at a Linux specific file system (ext2, ext3). So I reserved 3G space for building cross development environment or running big automatic compile script.

3. Leave the rest to store data only and make this partition dual accessible by both Linux and Window OS;

It's better to decide your directory structure and don't change it often even when your computer changed. It's easy for backup and maintenance.

2. Windows

Recommended applications to be installed for Windows OS.

Full set of Microsoft office ;

Adobe Acrobat;

MyIE2 (browser);

Ultraedit;

Antivirus serious;

金山影霸，RealOnePlayer, 暴风影音。

download tools: nettransport, cuteftp,

Quick Screen Capture;

System maintenance: System mechanic or Windows optimizer;

3. Linux

Linux is most installed by IT professionals to investigate, it is still weak for common desktop applications.

I usually use all the default packages for desktop usage.

For office document, use OpenOffice.

If you are a far east user, please refer to http://www.opencjk.org/projects/wineinput/index.html

If you are an IT professional and want to investigate some advance usage, please refer to

a. build a cross development network at Embedded section;

b. Setup SOHO network by Linux.

4. SOHO

Many sales people may approach you to sell their product for you SOHO network. I prefer to use Linux for all of these, although it usually means lack of technical support from suppliers. Actually you could find almost all solutions from internet.

The following configuration are essential for a software development based company.

a. networking. It is recommended to get a dedicated low-end router plus several switch although you could config one Linux machine as the router.

b. Define IP address range, bind your IP address with MAC address so it is easy to manage.

   Always use DNS to assign the IP address.

c. Config IP tables or your router may have some way to have firewall configured;

d. Setup NIS (Yellow Page service) so that all your intranet use can use the same account regardless of where he logged in;

e. Setup quota limit for every user accounte.

f.  Setup a common temp file for all users so that they can exchange huge file even it exceeds the quota limit.

g. Monitor your network if necessary.

h. backup all your data hourly, daily, weekly, monthly.

This page has links to some useful online resources for configuration that I have been used.

They could be used for setting up either desktop environment or embedded development environment.

How to setup J2EE, J2ME for Mobile Applications?

How to run IIS in Windows XP Home Edition?

How to install a Linux Virtual PC?

How to use Chinese in Linux?

How to setup WinCVS with SSH?

How to create a Linux daemon?

How to run XWindows Applications Remotely?

How to use ramdisk for Linux?

How to do DiskOnChip development?

How to config Apache?

How to debug Linux kernel?

How to use cgi-bin?

Definition of Real-Time OS

A lot of people think Real-Time OS means fast, actually, RTOS means the process deadline can be met generally (soft real-time) or deterministically (hard real-time). An RTOS is valued more for how quickly and/or predictably it can respond to a particular event than for the amount of work it can perform over a given period of time. Key factors in an RTOS are therefore a minimal interrupt latency and a minimal Process (or thread, which is different) switching overhead.

This is one of the OS chart I created to explain OS task switch.  It helps to explain time-slice based OS task switch, with interrupt support.

In general, there are two design approach associated with this. Preemptive OS and Cooperative OS, A preemptive multitasking OS will interrupt a running process when its time-slice is up (or any other reason). Cooperative multitasking, on the other hand, relies on the process itself to be nice and hand over control to other processes when it does not need it. In the latter case, a poorly designed application can easily monopolize the entire machine.

Project List

As the time goes by, I am afraid that I will forget what I did in the past, so I list my past projects for reference.

Project Name	Duration	Location	Responsibility	Project Summary
DS2000 CPR	24 months	Guangzhou China	Developer	Develop power on self-test program for a dual write processor Board, and conduct the test.
DS2000 DSW	12 months	Guangzhou China	Team Leader	PCB layout design with Mental Graphic Tool.
Telecom Training	12 months	All over China	Presenter	Pre-sales support, present telecom infrastructure products.
DORMIA	24 months	Singapore	Sole Developer	A distributed system for mobile information access, with multicast middleware support.
cdma2000 RNC	15 months	Singapore	Key Developer	Embedded Software /Object Oriented Modeling/C++ coding/Telecommunication domain, to build Radio Network controller (an element for CDMA infrastructure).
UMTS CLPc	7 months	Singapore, Dallas-Fortworth	Team Tech Leader	Another Software based element for CDMA (UMTS) system, in Telecom System/HAP Platform/C++ coding domain.
ISTUE	6 months	Singapore	Project Manager	Integrated Set-Top Box User Environment, a prototype on Windows Component Based architecture for feature development.
CDMA2000 System Integration	6 months	Chicago	Feature Leader	3G Wireless Telecom System Integration Lab integration and site integration support.
Internal BTS Router	8 months	Singapore	Team Test leader	Embedded Software Element/Transport Layer/Real-time Linux to implement basic IP Routing functionality.
Insec	6 months	Beijing, China	Department Manager	IP Security/Real-time Linux, port Freeswan to MPC860 environment and add new security feature to Linux kernel.
XMRadio	1 year	Singapore	Project Manager/Architect	A software component to handle US Satellite Radio Receiver.
CMMi Process Improvement	5 years	Singapore	Contributor	A lot of activities to identify and improve CMMi level for two global firmware organizations, from various CMMi perspectives.
Software Entropy Reduction	3 years	Singapore	Contributor	The aggregation of many activities to reduce software entropy of a codebase more than 1 million NCSL, and related to software production line to some extend.
Solid State Drive	2 years	Singapore	Contributor	Solid State Drive Controller.

Light-weight CMMi deployment

I was talking to one of my ex-supervisor in Motorola on his experience of deploying CMMi to another organization out of Motorola. There are some interesting findings about customizing CMMi for different organization cultures.  As Motorola has officially announced closing down of its Singapore Software Center, now we can comment a bit on their CMMi process, and what other organizations can learn from it.

Motorola is famous for the CMMI level-5 deployment to its various software centers across the world. These software centers are such process oriented organizations that low to mid management level take it for granted that the software requirement, design, implementation and testing document must be in place, and process audit must be in place. There are plenty of diagrams, analysis reports generated that becomes over-complicated, for good and bad.

In early 1990s, Software was such a special technical skill that only a small group of elite people can do it well, so Motorola setup various software centers around the world, with software process deployment initiative. Both the organization model and the process proved to be quite effective initially. Later as the software complexity grows, software process also grows to ensure the quality and efficiency of the code, which is excellent. However, these centers are far away from the end customers. The marketing team cannot transfer their pressure to software developers effectively in this organization structure.  The mid-level management turns to process focus, instead of customer focus, because that is the instruction implied by the evaluation process (for sure top management still emphasize they are marketing oriented).  All software projects are internal, which are quite easy going. When the profit margin of software business goes down and the company eco-environment got worse, eventually the company cannot afford this over-complicated business model and close down the center. 

This does not mean it is not possible to marry the good merit of process-oriented approach to innovation-oriented approach. One compromise way we found is to simply the CMMi process and deploy it implicitly. More specifically, requirement and software configuration management are the two most important KPAs a software organization needs to pay attention to, next comes implementation and test define. i.e, to implement what you are required to implement, and test what you are required to test. By telling your customer about implement customer requirement, and control your software team so that the entire team works towards the same goal, instead of working against each other (not personally, but because either Architect or process has flaw, I have seen a lot examples that people work against each other.),  you have the CMMI corner stone lay out.

The process effort should not exceed 5% of the total project effort. If there is no full time process team, then do 2 things:

1. have one internet-savvy young engineer to update and maintain an intranet to publish requirements (it is important to put requirements, designs, … into a common repository). 

2. List the key process flow in one Page (A4), make sure the font size is readable, let the team has free access to it, and have an experienced Engineer to audit regularly (biweekly or monthly).  A white paper (such as Software Development Manual) more than 2 pages is only useful for process team, nobody in the project team will read it. So DON’T waste your time.

In summary, the ultimate goal for an organization is not to reach certain CMMi level (though it is a side output, necessary for many human performance evaluation), but to make the final output predictable and quantifiable. Process is like lubricating oil, you won’t feel it if your machine running smoothly, and it is actually performing its duty implicitly. The most important thing: Software Process must come from real-life experience.  There are endless documents out there, people won’t convinced if you are only repeating a book story.

Why Embedded

Linux now spans the spectrum of computing applications for embedded world and there are endless articles on internet about embedded Linux. This article is the collection of some thoughts while teaching embedded Linux at Beijing University and lead related project. Hope it could be helpful for people with some computing background to ramp up on this domain.

Why Embedded?

The computers used to control equipment, otherwise known as embedded systems, have been around for about as long as computers themselves. They were first used back in the late 1960s in communications to control electromechanical telephone switches. Thousands Chinese engineers was working on digital exchange development in early 1990s, I was fresh at the time and lucky to be one of them. I was writing some boot code for 8086 board and play around with Logical analyzer everyday, but did not know the word “Embedded”. Only after several years working, I suddenly aware: this is embedded.

As the computer industry has moved toward ever smaller systems over the past decade or so, embedded systems have moved along with it, providing more capabilities for these tiny machines. Increasingly, these embedded systems need to be connected to some sort of network, and thus require a networking stack, which increases the complexity level and requires more memory and interfaces, as well as, you guessed it, the services of an operating system.

Off-the-shelf operating systems for embedded systems began to appear in the late 1970s, and today several dozen viable options are available. Out of these, a few major players have emerged, such as VxWorks, pSOS, Neculeus, and Windows CE.

How to evaluate your firmware debugging environment?

It is quite common that embedded system developers spent more time figuring out their environment before they can actually test-out their code. Therefore, it is vital to select the right tool in all stages of the entire life cycle. This article tries to summarize the experienced gained on embedded system development environment.

1. Get a good external/internal debugger.

2. You may not have the luxury to connect to an external debugger (for example, it is at factory production, no space to sold a JTAG connector), then use the ancient time method: print out to serial port;

3. You may not have the luxury to print out due to performance reason, but at least keep an UART connector, intrude the code to dump trace to flash or even DRAM, and find a way to extract later. If it is flash, you are so lucky that you can take out the equipment and analyzer somewhere else. I like it.

4. Logical Analyzer can be used to debug CPU execution indeed, provided that you have the corresponding Pod and software package. The distinct advantage is it can trigger/capture with external signals. JTAG can help but not as comprehensive as Logical Analyzer.

What else?

How to design a internal debug environment and exception handling system is a big topic? You are welcome to discuss the general rule of "good" debug-ability?

An ARM CPU mode diagram

I had several rounds of discussion with ARM support Engineers and have created a PPT slide on ARM mode. Thought it will be very helpful to understand the difference between various ARM CPU modes, and how to switch from each other.

Can you figure out the difference betweem ARM IRQ and FIQ from next diagram?

A metaphor for distributed system

There are many official, or rather impenetrable explanation of distributed system. When I was studying it in NUS, it really took me sometime to understand it.

Actually, I think the distributed system can be easily explained by comparing to our traffic system. Every vehicle is a self-controlled system, with its own thinking and follow certain rules, and the entire system works well. A traffic accident is either because the law is not good (very unlikely), or because some unit break the law, then the entire system has someway to recover.

WYSWYG Embedded Programming

WYSIWYG programming is a popular term used in PC software world, such as web design or PC object oriented programming. I am a big fan of tool aided code generation for embedded world, more specifically, use actor incarnation and Finite-State Diagram as the example below.

BTW: Please note Object Oriented style of programming does NOT mean Object Oriented Language. I could not find a suitable term for it, so I call it “WYSIWYG Embedded Programming”. 

An example of Actor diagram (instance of Object).

Protocol (structure definition) and FSM (dynamic behavior) example.

The question here is about reliability. You are comfortable to browse a web page generated through WYSWYG. If you are driving a car with auto collision-avoidance controlled by auto-generated code, do you trust the firmware or the human?

My answer is WYSIWYG tool is more trust worthy. Same as in PC world, our embedded team is usually composed of Engineers from all levels and various backgrounds. The great advantage of WYSIWYG is that WYSIWYG allows the user to visualize the target break down static and dynamic behavior, and even step/debug graphically. It not only helps the developers to gain better understanding of the architect, but also enables the team members to collaborate more efficiently based on their better understanding. Especially if there are many new team members.

With the structure breakdown, it is easier for developers to design the code to cover more comprehensive cases, and find out more corner cases for testing. There are cases that non-scalable architecture eventually makes the project next to impossible to have one human to oversee all logical paths,  and costs enormous amount of effort to do repeat but not efficient testing.

WYSIWYG may have impact on Performance and code footprint overhead. This is another solvable issue. There are various efforts in industry tackling this, which worth to discuss separately.  In fact, my experience shows human factor may pose more overhead.

A Light-Weight SCM site

There are plenty of resources regarding Software Configuration Management on the web now. However, normally a development team really need is only several key ideas of how to manage their codebase or documents, and limited number of document templates to start with.  Some team even cannot afford to have a full time SCM manager. It is not necessary to study the theory of SCM through.

SoftewareCM.org is created to help beginners to start within several hours. There is also a full vendor and resource sites list, plus remarks no more than 3 sentences. I came across this site occasionally. They have a good start point.

Software Configuration Management

I am heavily involved into the global effort of one big MNC to improve the Software Configuration Management at corporate level now. This situation reminded me that I should create a reference article on my understanding of software configuration management, and all the tools I have used, like CVS, ClearCase, SourceSafe, Telelogic Synergy and Perforce.

1. What is Software Configuration Management

Roger Pressman, in his book Software Engineering: A Practitioner's Approach, says that software configuration management (SCM) is a "set of activities designed to control change by identifying the work products that are likely to change, establishing relationships among them, defining mechanisms for managing different versions of these work products, controlling the changes imposed, and auditing and reporting on the changes made." In other words, SCM is a methodology to control and manage a software development project.

I think the modern SCM has become a concept interference with the development process and software architect. But first of all, let's start with the conventional (or tradition, standard, basic, or whatever you want to call it) concept of software configuration management - to manage the software variation over time. In other words, to version control the code, tracking/managing the change with label/version number, and reproduce the release upon request.

2. SCM vs. other Software Engineering Items

SCM is closely related with the development process and the Architect of the software system. More can be found at http://www.softwarecm.org/

Software Engineering

I like the definition of Software Engineering at Wiki: Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software. If you ask one hundred Software Engineers about the scope of software engineering, you might get 100 different answers. In my view, there are really 2 subcategories only:

1. How to present the software design? In other words, Software Modeling, or more concrete word, graphical representation of software blueprint;
2. How to make software engineers collaborate towards the same goal (instead of work against each other, don’t laugh, it happens). Or in other words, how to manage the space or time diversity of a software product, and how to break down the complexity of a software/firmware product (modularization).

The era that a single software hero creates an entire industry successed software product has gone. OK! Don’t argue with me that Linus Torvaldes is maintaining the Linux kernel by himself (still with other people’s help). There is only one Linus Torvaldes in the world, you and I are not as smart as him, I am talking about what’s happening in the industry. The industry trend is really how to manufacture software product through collaboration effort of teams made up of normal people, and located across the globe. Software architects and managers should really concern on making software implementation work and resource allocation more predictable. Has been saying that, they should really think about the above 2 Software Engineering questions.

In my opinion, these 2 questions could be expanded to a lot more questions as below. There are tons of books to discuss these questions. However, I merely list the possible questions for reference.

How to present the software design?

1. What is the most efficient way to model a software system? Why this modeling important?
2. What is the suitable tool for software modeling in your project?
3. Is it possible to auto-generate the code or code stub?
4. How to bridge the gap between business requirement, software design, implementation, testing and support? In other words, what is the best way to represent the software requirements?
5. How to control the auto-generation overhead? If the auto-code generation overhead is not acceptable, is there anyway to do it manually, retain the graphical design and write the efficient code manually?
6. Is it possible to test before implementation? Sounds weird? However, it is possible to just test code stub, which will be very useful for design validation at the early stage.

How to make software work predictable?

1. How to design software components or improve the modularity of existing codebase?
2. How to control the time variation? This comes to the topic of Software Verstion Control Management.
3. How to control the space variation? In other words, software reuse or variation based software reuse.
4. How to estimate software effort based on software components?
5. How to predict the software productivity based on Engineers capability and component complexity? Subsequently, how to make a realistic software schedule.
6. How to make test more effective? In other words, how to setup an effective test environment which is capable of generating many corner cases. How to quantify and improve test coverage?
7. How to make new Engineers (or all Engineers) easy to ramp up on one project?
8. How to break down the product scope and make project risk more predictable?