NWAPW Year 2+3

Technical Topics

Part 8: Software Components

In last year's pedestrian project we identified several distinct software components which we distributed among five small teams for implementation. We can do that again, at least initially...

Part 8a: Hardware Interface

We are using the same camera as last year, so most of what we learned last year -- reassembling the pixels from the RGB bytes in the Bayer8 data, despeckling, presenting the image in a window for the rest of the world to see -- can be recycled. Last year we looked for a half-dozen posterized solid colors to eliminate the effects of shadows, but this year we need to go the other direction, initially toward a luminance-only image from which to extract the boundary white lines. If you are wildly successful, you might have time to try to infer virtual white lines from a boundary between uniform pavement and the visually noisy off-track stuff like parked cars or vegetation. We can call this the initial image processing.

At the other end of the processing chain we have two kinds of output, presenting the same video image in a window with added annotations so we can see what your software thinks it is doing, and more importantly, actually controlling the car (steering it and setting its speed). Last year Peter did the visual integration but he may have other obligations this year, so you should think about image post-processing as a separate and important software component.

We have some example code on the LattePanda forum for using the Firmata APIs for controlling servos, but determining the proper timing and settings to get the hardware servos to behave consistent with our intentions usually involves substantial trial and error. Servo control might be a software component in its own right, and if you do it well, you will have a valuable skill for a career in robotics or other hardware/software interface fields in the computational universe. If you hope to scale this project up to a larger car with more inertia, there will be a response time lag between the commands issued by the decision-making software components and what the car actually does. You can start to prepare for that kind of project trajectory by emulating a response envelope in your servo control in this small model where the response otherwise seems effectively instantaneous.

A really ambitious group might replace the Firmata code in the Arduino with your own PWM generator (Custom Arduino code), but you would need to learn a lot of specialized tools, which could be a valuable skill in its own right.

Part 8b: Track Following

Finding the middle of the lane in the visual image is pretty easy. Deciding which parts of that middle line to pay attention to in deciding which way to steer the car requires analysis and experimentation. Whatever your camera might see very close to the car cannot have any useful effect on steering the car, because you don't have time to respond to it before the car has passed that part of the track. Similarly, distant parts of the image are irrelevant to steering the car, because you may never get there if the track turns away. You need to build a software model of which parts of the middle line are most relevant, and a weighting scheme to set the steering analysis appropriate to it, so that discontinuities in the middle-line data do not confuse your car.

Part 8c: Time and Speed

Your first efforts to steer the car will probably leave the speed control manual (using the radio remote). That way, if -- maybe I should say when -- something goes wrong, you can stop the car before it crashes into something (or somebody) simply by releasing the trigger on the remote control. Then you can program the car for a relatively slow steady speed, slow enough so somebody can walk along side and pick the car up off the ground before it crashes into something. When you have a high degree of justified confidence in your steering control, you can ramp up the velocity.

The the car speed increases, you will begin to notice the effects of how long it takes your software to process the incoming image before you tell the steering wheels which way to turn. You cannot do sharp turns on the track at high speed, not only because the car might spin out or roll over from lateral acceleration forces, but also because your software cannot respond quickly enough. This is why race tracks have no sharp turns. As you extend your car control towards higher speeds, you need to adjust the planning that goes into steering it to accommodate the increased effects of latency. If your programs sees a sharp turn coming up, you probably want to decelerate the car going into the turn. Speed analysis is an important part of making your car more sophisticated.

Part 8d: Developmental Support

Last year one member of the group maintained a GitHub repository for the project code. Somebody needs to be responsible for getting that up and maintaining it again this year.

The LattePanda has Wi-Fi hardware built in, so you might find it convenient to build a Wi-Fi remote interface to view what your software is seeing and doing on other computers connected to the internet (like your laptops). You could also use such an interface to start and stop the car. This is not essential to driving the car, but a great convenience, and seeing in the auditorium projector what the car sees in real time makes a compelling presentation on the last day.

Part 8e: Dividing into Work Groups

As I see it, these are the essential software components discussed above, which you should staff appropriately with team members (the links refer back in this page to where the terms are discussed):
1. Initial image processing

2. Steering analysis

3. Speed analysis

4. Servo control

5. Image post-processing

These are some tasks that may or may not deserve separate working group support, but you should think about:

6. GitHub repository

7. Wi-Fi remote interface

8. Custom Arduino code

If we have enough participants, there are some interesting tasks that can be worked on in parallel, then integrated after the basic car functions work properly:

9.  Adjusting the speed for the processing latency (slower for tight turns, faster for straight)

10. Predictive steering, so the car drives more smoothly

11. Detecting a stop sign and stopping at the right place

12. Inferring implied lines from parked cars or other non-traffic lane objects in the scene

13. Staying an appropriate distance behind another car in the same lane

14. Making turns (from a script?) when there are choices

15. Passing another car in the road (probably need extra hardware, like another camera)

16. Parallel parking (probably need extra hardware, like more cameras)

Any questions or comments? This is your project.

Tom Pittman

Rev. 2018 March 1