Air traffic flow management (ATFM) is a complex activity that is affected by many factors, including air traffic safety. One of the main issues in ATFM is the air holding problem (AHP) that occurs when aircraft are subject to restrictive flow control measures. These actions are taken while in flight, normally by circling in certain air regions or reducing aircraft velocity. This could be because of an airport closure, reorganization of air traffic or congestion in airspace sectors.
These activities are carried out by air traffic controllers, who are responsible for assessing the air traffic situation, analyzing possible risks, estimating the impact of possible restrictive measures and applying the measures that are deemed the best. Regardless of professional experience, performing these tasks in a small time interval without support from a specialized tool increases the risk of human errors and reduces the system's level of safety.
Intelligent systems can aid the air traffic controller in decision-making. These systems are able to analyze vast amounts of information, suggest possible actions and estimate their impact by forecasting traffic levels. This support allows the air traffic controller to assess the current air traffic situation; analyze possible restrictive measures that can be applied and suggested by the system; verify if the impact, which could be at the local and/or the overall level, for the chosen action is acceptable; and make the best suggestions of restrictive measures.
The system is also capable of learning from the decisions that have historically been taken by the air traffic controller, suggesting solutions that closely resemble those of the professional. The ability to learn from every air traffic controller is a very important feature because it makes the learning process and knowledge propagation faster and more in-depth.
The human capability to handle complex problems of this range is relatively limited. In critical situations, it is difficult to take actions without the support of a computer tool. The use of these tools is now more important than ever in assisting air traffic controllers in their decision-making, so they can perform their activities in a safe manner.
This paper presents a model that attempts to resolve the AHP in the ATFM environment in Brazil. The proposed solution was applied in two case studies in the Brazilian airspace. The first one is comprised of two flight information regions (FIRs): Brasilia (FIR-BS) and Curitiba (FIR-CW). The second case study is comprised of FIR-BS and the Recife FIR (FIR-RE). A FIR is a subsection of the airspace that is the basic unit of air traffic management (ATM) and control.
The proposed Air Holding Problem Module (AHPM) uses multiagent systems and reinforcement learning (RL) by applying the Q-learning algorithm for the learning process . AHPM is developed with the capability to suggest more precise and realistic actions based both on past experience and on human specialist knowledge and forecast the impact of restrictive measures on the local and/or overall level, thus increasing air safety.
This paper is organized as follows. Section 2 presents a literature review on the leading ATM problems and the approaches used for their resolutions. In Section 3, several air traffic management concepts related to this research are presented. Section 4 presents the methodology used to solve the AHP. Section 5 presents the proposed solution called the AHPM. Section 6 presents the case studies and their results, and Section 7 contains the final considerations.
2 Literature Review
Air traffic flow management problems arise from non-adherence to flight schedules, resulting in the possibility of congestion and inefficient use of airspace capacity [2-4]. Non-adherence to the flight schedule can be caused by delays in airport operations and a number of meteorological conditions.
Restrictive measures based on flight plans and empirical analysis by air traffic controllers do not take into account the actions' impact may have on neighboring sectors and/or FIRs . For this reason, it is not possible to perform a more accurate and in-depth estimation of the impacts that will occur before the air traffic controller can act. In some situations, problems may even be seriously aggravated, whereby the attempt to solve the original problem generates other, perhaps greater, problems.
According to Crespo et al. , FIR-BS, which is responsible for the central piece of the Brazilian airspace, manages approximately 50% of the air traffic volume in the country. The scale of the impact that failure to take into consideration all necessary factors could bring about is thus obvious.
Air traffic management is a complex activity. To effectively find a solution, the main problem can be divided into ground holding problem (GHP) and AHP [7, 8]. It is still possible to further divide it into subproblems: distance between airplanes en route, ground delay and route redefinition [].
The concept used by Agogino and Tumer  in addressing the solution of problems related to air holding, through the application of restrictive measures by varying the distance between airplanes en route, can be used in some simple scenarios. However, in more critical situations involving congested or saturated sectors, the results have not shown to be satisfactory . Because each sector is assigned one air traffic controller, its capacity is determined by the number of aircraft where a controller can handle safely and effectively. In Brazil, a sector is considered congested when it contains more than 11 aircraft and is considered saturated with more than 13 aircraft. Sectors are defined, in more detail, in Section 3.
Applying these measures will only delay congestion in the sectors or only reduce it in that particular moment, because the impact of the measure in a particular moment is not taken into account. When applied, it is possible to organize the line of aircraft en route in every sector, but this does not guarantee, for example, that increasing the distance between the aircraft will reduce congestion in a certain sector without impacting other sectors, thus acting without any control. This impact can be controlled when actions are taken to reduce saturated sectors or congested ones that are likely to become saturated.
2.1 Decision support systems
Decision support systems (DSSs) are defined as systems that support the decision-making process providing relevant information, suggestions and forecasts, based on actual information, to provide an estimate of the future situation, allowing certain actions to be taken in the present.
The automation processes must be carefully chosen, such as activity control, fault control and analysis, research and planning. DSS allows us to use data and models related to a domain of interest to resolve semistructured and unstructured problems [10, 11, 21-23].
According to Agogino and Tumer , it is essential that air transportation support systems be designed to offer means for automated and flexible management to meet the inherent needs of this kind of management.
One of the issues of ATM systems is the lack of explanation about the reason for a certain suggestion to the air traffic controller for decision-making. The reasons for some suggestions are not sufficiently clear to be accepted by the air traffic controller, as a result of the high complexity involved in the analysis of scenarios and the evaluation of these possible suggestions.
Among the concepts presented in the literature [5, 6, 11, 12, 17, 19, 20], DSSs can be classified into four groups by their way of operation:
- No autonomy: The system presents information and the human specialist must decide what is useful for every situation;
- Total autonomy: The system, which holds previous knowledge, analyzes every situation and makes the decisions;
- Semiautomatic (more automatic): The system holds intelligence to evaluate several situations, makes the decision in most situations; however, in some situations, it will require the human specialist to take the decision;
- Semiautomatic (more human): The system holds enough intelligence to analyze situations and present solutions to the specialist, who will ultimately decide what needs to be done.
In this research, the fourth concept—semiautomatic (more human)—is used. The system analyzes the situation and makes suggestions on actions to be taken by the human specialist, that is, decision-making is up to the air traffic controller. This choice is because of airspace safety concerns, so the air traffic controller will have the information generated by the system at his disposal while having total autonomy to choose between implementing the suggestion presented by the system or a new action based on his/her experience.
2.2 Multiagent systems concept
Multiagent systems consist of agents who interact in an organized fashion to obtain a certain objective. This kind of system is used with highly complex problems so that autonomous agents can find a solution by dividing the original problem amongst several interacting agents who have to meet their objectives. These distributed objectives are part of the main objective for success of the system.
There may be one or more agents interacting in a given environment. In some cases, agents can act on their own while in others, they can communicate with others. In more sophisticated systems, it may be necessary to add an agent whose sole purpose is to manage the other agents in the environment.
The work of Agogino and Tumer  used a multiagent system architecture with the Future ATM Concepts Evaluation Tool (FACET), to improve the results . FACET is an ATM tool based on the US airspace. It is actually being used by the Federal Aviation Administration and the National Aeronautics and Space Administration. The authors identified three items to be studied and obtain the necessary improvements:
- The impact of actions of two new agents: the delay of aircraft on the ground and the redefinition of routes and distance between aircraft;
- The impact of coupling between the agents' actions;
- The benefits of providing rewards for the agents using previously calculated evaluation functions.
Figure 1 presents the interaction between agents to take restrictive measures in order to prevent congestion in a sector.
It is possible to verify how a multiagent model can solve some specific problems of air traffic flow. In this model, there are six agents that communicate among them to avoid one sector with possible congestion. Thus, when the aircraft are input in the model, it analyzed possible risk situations and issues that need to be solved. In this case, two agents will suggest the reroute for some aircraft and reduce the possible congestion.
Various interacting agents can be identified: ATFM; airport terminal management and the integrated management of the FIRs and others . Thus, the notion of identifying and organizing agents in this structure and making them interact to achieve the best results has become the main focus of research in the field.
2.3 The reinforcement learning concept
Reinforcement learning (RL) is the process in which an agent, without previous knowledge and in a predetermined number of interactions, interacts with the environment to achieve its objectives and, in doing so, is rewarded for its actions and objectives achieved.
All the decisions taken by the agent take into account the history of actions previously executed to achieve an optimal result. One of the fundamentals of RL is that the reward for every action is such that the agent is stimulated if great results are achieved and discouraged if the results would be worse than previously obtained. In this way, the agent can always look for the best action to be taken based on past actions, while trying to improve its results.
According to Agogino and Tumer , the use of agents can improve the performance of a system that supports decision-making for the air traffic controller up to 20%—the proportion of air traffic controllers who accept the suggestions made by the system. According to Souza et al. , the restrictive measures taken are not stored in a proper way to utilize the best practices, in order to support decision-making in similar situations.
A relevant contribution of the studies that adopts this concept in the recommendations based on analysis of previous decisions taken, making it possible the system to learn with air traffic controllers on a daily basis [6, 11, 16].
3 Air Traffic Management
Air traffic management (ATM) focuses on providing means to manage air traffic, taking to account factors such as safety, fairness, weather and financial factors . ATM is necessary to monitor airspace, controlling the flow of aircraft in predefined airways, and it can be managed in an integrated way, thus obtaining greater effectiveness with made decisions.
ATM can be divided in three groups: airspace management, whose goal is to improve airspace to meet the demand without physical expansion of the system; air traffic control, which focuses on controlling the aircraft traffic, providing the mandatory information to maintain flight safety; and ATFM, which focuses on providing information for the environment, so aircraft flow can be safely maintained and the impact of unforeseen situations at other levels be minimized. ATFM at the tactical level is the main focus of this study.
ATFM can be further divided in three phases:
- Strategic level: the strategic planning for flights including the time from 48 to 2 h before the flight;
- Pretactical level: tactical decision-making, including the time from 48 to 2 h before the flight; and
- Tactical level: tactical decision-making, including the time 2 h before the flight until it reaches its destination.
3.1 The air holding problem
The AHP occurs in ATFM when aircraft are placed on hold while in flight. The reason could be the closing of an airport—for example, in the case of adverse weather conditions or terrorist actions—or because of excessive traffic in a certain airspace sector.
When an action is chosen for a subscenario, the result of this action can improve one subscenario and make others worse, in such a way that the overall scenario can turn into a high-risk problem for the safety of all involved. Some studies that seek to resolve this problem focus only on the moment of arrival of the aircraft and the necessary actions to reduce the impact, while others focus on the interaction of intelligent agents acting in certain areas with the human being [11, 15, 18].
3.2 ATFM in Brazil
The Brazilian airspace is comprised of the airspace above its territory, in addition to an area over the Atlantic Ocean. Figure 2 illustrates the Brazilian airspace, divided into five FIRs: FIR-Amazônica (SBAZ); FIR-Recife (SBRE); FIR-Brasília (SBBS); FIR-Curitiba (SBCW) and FIR-Atlântico (SBAO).
The FIRs are divided into control sectors for better flow management and greater control. Currently, there are 46 control sectors: 14 in SBAZ, 8 in SBRE, 12 in SBBS, 10 in SBCW and 2 in SBAO.
The sectors are under supervision of the air traffic controllers stationed at the area control center. Each FIR is under the responsibility of an area control center, which is responsible for delivering area control services, providing information and supervising aircraft both en route and in the act of landing or takeoff. Each sector is assigned one and only one air traffic controller; however, during low-demand times, one air traffic controller may be assigned two or more sectors. This organizational structure makes it possible to maintain minimum separation between aircraft, as well as keep the air traffic flow in an organized way in its respective airspace. The number of aircraft in a sector has a direct influence on the management complexity, that is, the more aircraft in a certain sector, the greater the safety risks in the ATM .
The focus of the study is on the concept of critical scenarios, that is, when more aircraft are flying and more sectors are saturated or congested. These scenarios are currently of special interest to Brazil because the country is preparing to host two large international sports events during which air traffic volumes are forecast to increase dramatically. For this reason, dates, times and airports with high air traffic volumes were chosen for this study.
4.1 Evaluation scenarios
The case study was performed in the Brazilian airspace, involving three of its FIRs: FIR-BS, FIR-RE and FIR-CW. FIR-BS was chosen for being the most important and complex of the Brazilian airspace, including the busiest airports, and handling the highest volumes of air traffic. FIR-CW and FIR-RE were chosen because they comprise several regions of the country, making the case study more complete. Apart from that, FIR-RE has a considerable amount of over flights, allowing analysis of flights with no arrival or departure in the current FIR but with effect on actions to be taken. Figures 3 and 4 show the airspace used in the first and second case studies, respectively.
FIR-BS is divided into 12 sectors as presented in Figures 3 and 4. Among the airports in this FIR, seven were chosen: Brasília, São Paulo/Congonhas, São Paulo/Guarulhos, Campinas/Viracopos, Belo Horizonte/Confins, Rio de Janeiro/Santos Dumont and Rio de Janeiro/Galeão. FIR-CW is divided into 10 sectors as shown in Figure 3. Among the available airports, four were chosen: Curitiba, Florianópolis, Campo Grande and Porto Alegre. FIR-RE is divided into eight sectors as shown in Figure 4. Among the airports present in this region, six were chosen: Recife, Salvador, Fortaleza, Natal, Teresina and Maceió.
In the proposed model, the concept of analysis of adjacent sectors is used, even if these sectors are in a FIR outside of the experiment. This modeling strategy is of great importance because it prevents actions taken to resolve problems in a certain sector from resulting in severe traffic issues in an adjacent sector. For this reason, in each case study, all sectors from FIRs that were not part of the study but were adjacent to a FIR that was part of the study were also included. In the first case study, that meant including 23 sectors from FIR-AZ, FIR-RE and FIR-AO. In the second case study, 22 sectors were added from FIR-AZ, FIR-CW and FIR-AO.
Both case studies were carried out using flight schedules for October 7, 2011 and November 11, 2014. These days were chosen because they were right before national holidays, resulting in scenarios with higher air traffic volumes. The period between 17:00 and 18:59 was selected. This period was divided into four intervals of 30 min each to evaluate every scenario in a reasonable time.
4.2 Agent configuration
To perform the two experiments, an agent structure using FIR agents and a central agent was implemented. Both agent types are described in the succeeding texts.
Each FIR will have one agent that is responsible for providing suggestions on air traffic controller actions. Thus, the first experiment will have two FIR agents: FIR-BS agent and FIR-CW agent; in the second experiment, the agents will be FIR-BS and FIR-RE.
In the AHPM model, a central agent is responsible for the main control of the system. This agent will receive the suggestions for action from each FIR and evaluate which is the best for the overall system. After it has identified the best actions to be taken for every FIR, the central agent will verify if there are similar scenarios in the past and what actions were taken by the air traffic controller, including their results. This process is explained in detail in Section 5.
The FIR agent checks the current state of its FIR and, based on learning, looks for matching or similar scenarios that have occurred in the past and been registered with the central agent. The state of the FIR indicates if the air traffic is normal, congested or saturated. If better results are found, the agents will suggest the associated actions to the air traffic controller. If past actions cannot be found, the agent will determine the best actions to be taken based on the forecast of the new state of the FIR resulting from each possible action and present the best three to the central agent.
In case there were results that obtained an improvement in air traffic, the past actions will be presented as suggestions to the air traffic controller. If they are not present or achieve worse results, the first three suggestions from every FIR will be analyzed. The number of suggestions was chosen so that it can improve the agent's learning with time. The agent will then analyze all suggestions received from all FIR agents and select the one that has the best effect on the overall system.
Because the goal of the study is the ATFM decision-making process while aircraft are en route, the following restrictive measures will be considered in the two experiments:
- Delay aircraft en route;
- Forward aircraft en route.
The scheduled time for an aircraft to leave or arrive at a sector can be delayed or forwarded by adjusting the aircraft's speed.
5 Air Holding Problem Model
The proposed model, called AHPM [], was developed using multiagent systems and RL. The model is comprised of four integrated submodules as illustrated in Figure 5:
- Information Collection Module (ICM). ICM is responsible for storing information generated by the air traffic controllers. Comprising the historical database, it is stored in Flight Information database. The historical information is composed of the flight plans and its deviations, such as applied restrictive measurements, weather conditions, etc.
- Reinforcement Learning Module (RLM). RLM is responsible for system learning in two steps: (a) to receive information from the ICM and (b) to calculate the values for actions and states in every scenario by a given certain time instance and to use the knowledge obtained by the agent to achieve optimum results. These results will be calculated by evaluation functions defined by the Q-learning algorithm and stored in the learning database to provide the necessary information in an oncoming execution. This database contains applied restrictive measurements, a rule structure that makes it possible to work in a real-time environment, that is, current situation, and achieved results.
- Scenario Forecast Module, responsible for evaluating the scenario at a time instance Tn+1, to show air traffic controller what could happen in case a suggested action is accepted and taken. Scenarios will be forecast for both the local and overall levels.
- Decision Support Module, responsible for integrating all tasks in an online manner. The module will present all possible actions to be taken and all the resulting actions after the execution of a certain action, including the impact on other levels in a certain time instance. The impact of an action taken at one level in a certain time instance on other local and/or overall levels is estimated by this module.
5.1 Communication between agents
Agent learning is the process by which the agent, through time, will learn to suggest the best actions based on results achieved in the past. The proposed model is comprised of two agent categories:
- FIR agent: will perform the analysis necessary to choose the best actions for each FIR;
- Central agent: will analyze the suggestions made by the FIR agents and select the ones that will produce the best results at the overall level.
Figure 6 shows the communication mechanism between the agents of the proposed model.
The modules in Figure 5 are software components of AHPM, that is, a general structure to improve the utilization. The agents are parts of this software that are included inside of RLM. This module is responsible for the intelligence such as learning and communication of the AHPM, refer to Figure 6.
5.2 Evaluation functions
In reinforcement learning, the agent is rewarded according to the results achieved. That reward is defined by an evaluation function, which assesses the relative benefit of the action suggested by the agent compared with others that are registered with the system. Each agent has its own evaluation function, as described in the succeeding texts.
5.2.1 FIR agent evaluation function
The evaluation function is defined as follows:
- FFIR(s) is the sum of the congested sectors adjacent to sector s α is the number of aircraft in sector s, and β is the maximum number of aircraft in a sector over which the sector is considered congested, in Brazil, β = 11;
- CFIR(s) is the sum of aircraft in the congested sectors adjacent (S) to sector s;
- EFIR(s) is the evaluation function for the state of the FIR.
FFIR(s) will be calculated for each sector as illustrated in Figure 3. For example, FFIR(SBCW14) will check its adjacent sectors (SBCW-13, SBCW-03, SBCW-04, SBBS-01, SBBS-10, SBBS-11, SBBS-12), and CFIR(s) will sum aircraft in these sectors when it is congested. In other words, EFIR(s) will identify the state of FIR, that is, how congested FIR is.
The index of air traffic level was defined as the result of evaluation functions (FIR and central) and represents the level of total congestion in the system. The higher the index of air traffic level, the more aircraft are present in the Brazilian airspace.
5.2.2 Central agent evaluation function
The central agent evaluation function is to determine the collection of actions that have the minimum impact on the future overall level, based on the actions suggested by the FIR agent. The evaluation function is defined as follows:
- GCentral(FIR) is the sum of the FIRs that have at least one congested sector. α is the number of aircraft in sector s, and β the limit of aircraft for a non-congested sector, which is defined as 11 in Brazil.
- PCentral(s) is the sum of the congested sectors on the overall level, that is, all the sectors from the FIRs that were analyzed by the EFIR(s) function. α is the number of aircraft in sector s, and β the limit of aircraft for a non-congested sector.
- DCentral is the sum of the aircraft on the overall level, including all the necessary sectors for every FIR.
- HCentral is the evaluation function for the state of the overall level.
6 Case Study
As explained in Section 4, because the object of the study is the ATFM decision-making process while aircraft are en route, the following restrictive measures will be considered in the two experiments:
- Delay aircraft en route; or
- Forward aircraft en route.
Each experiment will be presented in three phases:
- Initial state (IS);
- Future state T1: the resulting state after the actions taken by the air traffic controllers;
- Future state T2: the estimated state after the actions suggested by the AHPM.
For every phase, the following steps will be taken:
- Evaluate the state for every sector;
- Evaluate the state for every FIR; and
- Evaluate the state at the overall level.
6.1 Traditional approach
The two case studies will be presented in two stages: the first one with the traditional approach carried out by the air traffic controller and the second with the approach proposed in this study.
6.1.1 FIR-BS, FIR-CW and overall level
Figure 7 presents the IS and the future period T1, which comprises three consecutive time intervals in FIR-BS, FIR-CW and the overall environment. It can be observed that there was improvement in time, but all states were congested. The index of the air traffic level, which represents the current state, determines how congested and/or saturated a specific scenario is, indicating where restrictive measures are needed.
In the FIR-BS, the air traffic level index was 463 in the IS and was reduced to 84 in the last period analyzed. However, all the states were congested during the analysis. The air traffic level index is the result of the evaluation functions (FIR and central), illustrating how congested the sectors are in a certain time interval. The closer the air traffic level index is to zero, the less traffic there is in one FIR or in the whole Brazilian airspace.
In the overall level, including FIR-BS and FIR-CW, it is clear that there was improvement in time; however, in the last interval, the state suffered a considerable increase in congestion. The air traffic level in the initial interval was 4260, which was reduced in the two following intervals to 1316 and 1280 respectively, but in the fourth interval, there was an increase to 2088 as a result of the increase in congested sectors. The closer the results of the evaluation are to zero, the better the conditions of air traffic environment, thus tending to no congestion if the index drops to zero.
Figure 8 presents the evolution from the IS to the future period T2, following the recommendations of the FIR agent in FIR-BS, FIR-CW and the overall environment. It can be seen that there was improvement in the second interval of FIR-BS when the index was reduced from 463 to 0, resulting in no congestion and a gradual increase in the next intervals. In the FIR-CW, it is clear that there was a gradual improvement, reaching no congestion in the third interval when the index reached zero.
In the future period T2 at the overall level following the recommendations of the FIR agent, considering FIR-BS and FIR-CW, that is, following the best actions for each FIR, it is clear that there was improvement in time; however, in the last interval, the state suffered a considerable increase in congestion as a result of the increase in congested sectors.
In the two experiments, the central agent will evaluate the three best suggestions from every FIR and choose which will yield the best group of restrictive measures to be applied in every FIR against the verification and projection of scenarios.
6.1.2 FIR-BS, FIR-RE and the overall environment
Figure 9 presents the development the IS and of the future period T1, which is composed of three consecutive time intervals in FIR-BS, FIR-RE and the overall environment. It can be observed that there was improvement in time. The air traffic level index, which evaluates the current state, was 463 in the IS and was reduced to 84 in the last analyzed period. All the states were nevertheless congested during the analysis. In the overall environment, taking into account FIR-BS and FIR-RE, it can be seen that there was improvement in time; yet, all sectors were congested.
The presentation of the results of the AHPM approach was divided into two parts. In the first part, the FIR agent chooses the best action for the FIR for which it is responsible. In the second part, the central agent receives the three best actions for every FIR and chooses the best action for the overall environment.
Figure 10 presents the evolution of the IS and of the future period T2, using the AHPM approach and the FIR agent in FIR-BS, FIR-RE and the overall environment. In the FIR-BS, it can be observed that there was improvement in the second interval when the index dropped from 463 to 0, eliminating congestion, followed by a gradual increase during the subsequent intervals. In the FIR-RE, it can be seen that there was a gradual improvement, having no congested sectors in the last two intervals. In the overall environment according to the AHPM approach and the FIR agent, considering FIR-BS and FIR-RE analyzing the graph, one can observe a considerable improvement from the first to the second interval when the index is reduced from 3400 to 329.
Albanese, M.S., & Mitchell, S. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues.Academic Medicine, 68 52–81.CrossRefGoogle Scholar
Allen, B.S., Otto, R.G., & Hoffman, B. (2000). Casebased learning: Contexts and communities. In S. Tobias & D. Fletcher (Eds.)Training and retraining: A handbook for business, industry, government, and the military (pp. 443–471). Englewood Cliffs, NJ: Prentice Hall.Google Scholar
Blumenfeld, P.C., Soloway, E., Marx, R.W., Krajcik, J.S., Guzdial, M., & Palincsar, A. (1991). Motivating project-based learning: Sustaining the doing, supporting the learning.Educational Psychologist, 26, 369–398.CrossRefGoogle Scholar
Bruer, J.T. (1993). The mind's journey from novice to expert.American Educator, 17(2), 6–15, 38–46.Google Scholar
Christensen, C. R. (1987)Teaching and the case method Boston: Harvard Business School.Google Scholar
Dahlgren, L.O., & Pramling, I. (1985). Conceptions of knowledge, professionalism, and contemporary problems in some professional academic subcultures.Studies in Higher Education, 10(2), 163–173.CrossRefGoogle Scholar
Ertmer, P.A., & Quinn, J. (1999)The ID CaseBook: Case studies in instructional design. Englewood Cliffs, NJ: Prentice Hall.Google Scholar
International Board of Standards for Training, Performance, and Instruction (IBSTPI, 1984),Instructional design: The standards, Washington DC: Author.Google Scholar
Johnson, S.D. (1988). Cognitive analysis of expert and novice troubleshooting performance.Performance Improvement Quarterly, 1(3), 38–54.CrossRefGoogle Scholar
Julian, M., Kinzie, M.B., & Larsen, V.A. (2000). Compelling case experiences: Performance, practice, and application for emerging instructional designers.Performance Improvement Quarterly, 13(3), 164–201.Google Scholar
Kemp, J.E., Morrison, G.R., & Ross, S.M. (1996).Designing effective instruction (2nd. Ed.). Upper Saddle River, NJ: Merrill/Prentice Hall.Google Scholar
LeMaistre, C. (1998). What is an expert instructional designer? Evidence of expert performance during formative evaluation.Educational Technology Research and Development, 46(3), 21–36.CrossRefGoogle Scholar
Lincoln, Y., & Guba, E. (1985)Naturalistic inquiry. Thousand Oaks, CA: Sage.Google Scholar
Lundeberg, M.A., Levin, B.B. & Harrington, H.L. (Eds.) (1999)Who learns what from cases and how? The research base for teaching and learnign with cases. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
McNergney, R.F. (1999). Foreword. In M.R. Sudzina (Ed.),Case study applications for teacher education: Cases of teaching and learning in the content areas (pp. v-vi). Boston: Allyn and Bacon.Google Scholar
Perez, R.S., & Emery, C.D. (1995). Designer thinking: How novices and experts think about instructional design.Performance Improvement Quarterly, 8(3), 80–95.Google Scholar
Quinn, J. (1994). Connecting education and practice in an instructional design graduate program.Educational Technology Research and Development, 42(3), 71–82.CrossRefGoogle Scholar
Rowland, G. (1992). What do instructional designers actually do? An initial investigation of expert practice.Performace Improvement Quarterly, 5(2), 65–86.CrossRefGoogle Scholar
Rowland, G., Parra, M.L., & Basnet, K. (1995). Educating instructional designers: Different methods for different outcomes. In B.B. Seels (Ed.),Instructional design fundamentals: A reconsideration. Englewood Cliffs, NJ: Educational Technology Publications.Google Scholar
Smith, P.L., & Ragan, T.J. (1999),Instructional Design (2nd. ed.), Upper Saddle River, NJ: Merrill/Prentice Hall.Google Scholar
Sykes, G., & Bird, T. (1992) Teacher education and the case idea.Review of Research in Education, 18, 457–521CrossRefGoogle Scholar
Whitehead, A.N. (1929).The aims of education: And other essays. New York: Macmillan.Google Scholar
Williams, S. (1992). Putting case-based instruction into context: Examples from legal and medical education.Journal of the Learning Sciences, 2, 367–427.CrossRefGoogle Scholar