Evaluation of information systems in health care:a framework and its application Hannu Salmela and Pekka Turunen Turku School of Economics and Business Administration Rehtorinpellonkatu
Trang 1Evaluation of information systems in health care:
a framework and its application
Hannu Salmela and Pekka Turunen
Turku School of Economics and Business Administration
Rehtorinpellonkatu 3, 20500 Turku FINLAND
The objective of this study is to develop a framework for assessing the costs and benefits of information systems
in health care The framework combines views from different disciplines, such as information systems evaluation, medical informatics, and health economics It suggests that the impact of health care information system should be assessed on multiple levels: the quality of medical information, the quality of diagnostic decisions and, ultimately, the quality of health care services To illustrate the application of the framework, an evaluation plan is developed for an information system called Computer Assisted Notification of Drug Effects
on Laboratory Tests (CANDELA) We assume that the framework and the evaluation plan benefits researchers and practitioners in the evaluation of similar systems
1 Introduction
While information technology (IT) expenditure in
hospitals is increasing, the effects of these
investments on health care services have not been
extensively studied (Ovid MEDILINE 1966-1997,
Ingasoll et al 1990; van der Loo et al 1995).
Although individual studies have suggested a
positive relationship between the level of IS
investments and the productivity of health care
services (c.f Menon et al 1996), the overall results
of IT investment profitability studies have been
inconclusive (Mitra & Karim 1996 p 29-31) On
the other hand, general IT investment productivity
does not guarantee the productivity of a single
health care information system
In many ways, the evaluation of information
systems in health care faces similar challenges as
the evaluation of IS in other types of organizations
Costs are often indirect and difficult to measure
Organizational impacts and benefits, on the other
hand, are often intangible and their realisation may
take a long time (Saarinen 1993; see also Ives et al.
1983)
Hence, the key principles of the framework
presented in this paper are derived from IS
evaluation literature Both the improvements in
information, improvements in individual
decisions/actions, as well as improvements in
organization level can indicate IS effectiveness and
success (DeLone & McLean 1992) The selection
between these measures is to a large degree
dependent on the values and objectives driving the
evaluation The type of the systems and the
economic considerations of gathering the
evaluation data need to be considered as well
There are, however, additional challenges in IS
evaluation in health care context Most notably,
information systems can leverage improved
treatment and consequently contribute to patients’ health Since the health impacts are difficult to evaluate in monetary units, the costs of the system have to be contrasted with improved "utility" or
"outcome", rather than with monetary benefits
To illustrate the application of evaluation measures,
an evaluation plan is developed for an information system called Computer Assisted Notification of Drug Effects on Laboratory Tests (CANDELA) The system encodes and links pharmacological drug interference information into a laboratory
information system (Grönroos et al 1995 a; Grönroos et al 1995 b; Grönroos et al 1997) It
assists physicians in interpreting laboratory analysis results and has potential both to reduce costs and to improve the quality of health care services The system is currently being implemented in the University Hospital of Turku Its systematic evaluation will start in the fall 1997
2 Evaluation of information systems
The evaluation of the effectiveness of an information system constitutes one of the key issues in information systems research In research, well-defined outcome measures are needed to ensure that the results from different studies are comparable They are a prerequisite for information systems research to make a contribution to IS practice In practice, success measures are needed
to evaluate IS practice, policies and procedures (DeLone & McLean 1992)
While a single measure of IS success or IS effectiveness would certainly be desirable, it seems unlikely that such a measure could be found Instead, research has provided taxonomies of success variables, which can be applied in different
situations (DeLone & McLean 1992; Grover et al.
1996) In general, the success of an information system can be evaluated through 1) The quality of
Trang 2information provided to the users, 2) The impact of
IS on users’ thinking, decisions or actions, and 3)
The impact of IS on organization level costs and
benefits
2.1 User sastisfaction
Due to the difficulty to assess IS impacts on
individuals or organizations measures based on
user perceptions have become prominent within IS
literature (Galletta & Lederer 1989) User
information satisfaction (UIS) is probably the most
widely used single measure of IS success The
original user satisfaction instrument contained a set
of 39 factors (Bailey & Pearson 1983) The
satisfaction of users was calculated as the sum of
user’s positive and negative attitudes to these
factors Other researchers have later developed
shortened and modified versions of UIS (Ives et al.
1983; Doll & Torkzadeh 1988; Saarinen 1993) so
much that in 1989, Miller identified 12 different
UIS instruments (Miller 1989)
Some researchers consider UIS as being related to
higher systems use, which in turn is related to
higher individual and organizational performance
(comp DeLone & McLean 1994; Grover et al.
1996; Scott 1994) However, user satisfaction
should be seen as a signal of acceptance of users
rather than a measure of organizational outcomes
While organizational effectiveness measures focus
on actual outcomes, user satisfaction focuses on
process Thus UIS is likely to be more useful in
finding critical problems of IS implementation or
use process than in evaluating the organizational
outcomes
2.2 Individual impact
Some researchers have tried to develop more direct
measures of the impact of an IS on users' learning
and decision making Ultimately, individual impact
should be measured on the basis of whether
information causes the receiver to change his or her
behaviour (Mason 1978) Although questionnaires
can be used here as well, many studies have relied
on laboratory tests where the impact of IS on
decision processes can be directly observed
(Dickson et al 1977; O'Keefe 1989) A limitation
of laboratory tests is that they fail to take into
account the real world environment where the
information system will be used (O'Keefe 1989)
2.3 Organizational outcome
Measures of IS costs and benefits have been more
common in practice than in research Academic
researchers have tended to avoid organizational
performance measures because of the difficulty of
isolating the effect of IS effort from other effects which influence organizational performance
(Grover et al 1996) In practice, ex-ante
evaluations have been more frequently while academics have preferred ex-post evaluation
(Parker et al 1988) For research purposes,
evaluation of IS cost-effectiveness is costly and inhibits comparisons between different studies
(Ives et al 1983 p 785-786; Saarinen 1993) Thus,
standard IS evaluation methods are needed in IS research
Expressing impacts in monetary terms places additional challenges on evaluation Some of the organizational impacts of information systems, such as improvements in products and services or improved management are often intangible Traditional accounting systems rarely provide the information needed to evaluate the costs and benefits associated with a particular IS (Matlin 1979) Hence, many researchers have proposed methodologies to estimate the actual contribution
of IS on firm performance (see Grover et al 1996).
DeLone and McLean (1992) conclude, however, that much work still needs to be done in this area
It appears that a comprehensive evaluation of an information system requires multiple measures (DeLone & McLean 1992; Saarinen 1993 p 51) Both the improvements in information, improvements in individual decisions/actions, as well as improvements in organization level can indicate IS effectiveness and success The selection between these measures is to a large degree dependent on the values and objectives driving the evaluation The type of the systems and the economic considerations of gathering the evaluation data need to be considered as well
3 Evaluation of information systems in health care
In many ways, evaluation of information systems
in health care is no different from that in other types of organizations
There are, however, additional challenges in IS evaluation in health care context Most notably, information systems can leverage improved treatment and consequently contribute to patients' health Because of the potential impacts on patients' lives more strict measurements are used to evaluate health care information systems
3.1 User satisfaction
User satisfaction has been used in evaluating health care information systems (for example Gardner & Lundsgaarde 1994) A number of studies have also applied UIS instruments developed in information
Trang 3systems science Pearson’s original user satisfaction
measure has been applied in the evaluation of
Hospital information systems (HIS) in 160
Veterans Administration Medical Centers (Bailey
1990 p 51) That study is further adapted in the
Clinical Computerised Information System (Pugh
& Tan 1994) Bailey and Pearson’s (1983) user
satisfaction measure has been applied into HIS
DSS (Dupuits & Hasman 1995) Chin and McClure
(1995 p 717-721) have used Doll and Torkzadeh’s
UIS instrument to evaluate clinical information
systems
The more strict approach to evaluation is, however,
reflected in the role that UIS measures have in
health care IS evaluation The role of user
satisfaction has not been as prominent as in general
IS research In health care context, only about 4 %
of the studies used user satisfaction effect measure,
whereas in IS research the fraction is 20% (van der
Loo et al 1994; Grover et al 1996) Hence,
although UIS is seen as providing insights into the
usefulness of the system as perceived by users, in
health care it is not very widely used as a surrogate
measure for systems effectiveness Instead, more
direct measures about the impacts on patients’
health are used Surprisingly, user satisfaction
measures have not been used in the evaluation of
supporting and auxiliary type of information
system either (see van der Loo et al 1994, p 50).
Such systems are used in making appointments,
and in managing documentation, financial
transactions and personnel information They
support patient’s welfare only extremely indirectly
(van der Loo et al 1994 p 47 and 50).
3.2 Individual impact
The impact of an information system on decision
making, particularly on diagnostic and treatment
decisions, has been a common basis for evaluation
(for example Maria et al 1994; Wagner & Cooper
1995) A typical evaluation design is based on two
groups where one group uses the information
system while the other doesn’t A recent study that
classified evaluative studies in health care found
that about 64 % of evaluations were based on two
group comparative study (van der Loo et al p 49).
3.3 Organizational outcome
In many cases, a demonstrated impact on treatment
effectiveness is considered as a sufficient criteria
for IS success It is seen as adequate justification
for widespread systems implementation and use
The value of health outcomes should, however, be
assessed in order to ensure economic use of
hospital resources and optimal health care services
for patients
In health economics, several concepts and methods have been developed to assist in evaluating the value of improved health outcomes The three main types of cost and utility measures used in health economics are: cost-effectiveness, cost-utility and cost-benefit analysis
In cost-effectiveness analysis the patient outcomes are measured in the most appropriate natural or physical units such as: life years gained, disabilitydays saved, points of blood pressure reduction Analysis is appropriate when a treatment has only one effect on patient’s health Such analysis can be supported with ratios such as cost per life-year gained or reverse life-years gained per
dollar spent (Drummond et al 1987).
In cost-utility analysis the patient’s health effects are expressed as quality adjusted life-years In cost-utility analysis even multiple health effects of treatment, not necessarily common to both alternatives, can be compared In cost-benefit analysis both the costs and health outcomes are
estimated in monetary units (Drummond et al.
1987)
Cost-benefit studies have been more frequent in the evaluation of IS in health care than in IS evaluation
in general In health care context, 13 % of the evaluative studies used cost-benefit analysis This
is still considerably more than in IS research, where
the fraction is only 4% (van der Loo et al 1994 and Grover et al 1996).
The cost-benefit type of analysis used in information system science have had only a limited affect on this type of evaluation in health care There are some studies in the evaluation of health care information systems where cost-benefit and cost-effectiveness terms have used from point of view of information systems science (Zielstroff 1985)
In information systems science intangible effects and costs are considered as a difficult problem (Saarinen 1993), but in health economics those things are not seen as insurmountable problems In information system science intangible costs and different types of effects are listed in evaluation reports (Johnston & Vitale 1988) But there are only few attempts to quantify intangible benefits and translate them into measurable comparable
values (like Money et al 1988) In this area more
models could be adopted from the cost-utility analysis in health economics They could facilitate comparison of several information systems Such analyses have not been really common in information science
Trang 4One difference between information system science
and health economics is that the cost and utility
analysis for information systems are more often
ex-ante than in health care In health care physicians
want to develop and give best possible treatments
The associated costs are thought only afterwards
4 A framework for evaluating medical
alert systems
A medical alert system is defined as an information
system which supports physicians in making
diagnostic and treatment decisions by providing
alerts based on information in clinical databases
(e.g patient information, patients medication
profile, drug information, treatment information)
The core of such system is a knowledge base,
which contains medical research results about
various interactions that need to be taken into
account in diagnostic or treatment decisions The
basic reason for using such systems is that an
individual physician can rarely master all possible
information of different interactions while making
routine clinical decisions
The evaluation of an alert system is a challenging
task The following framework attempts to provide
a comprehensive view of the costs and impacts of
such system Thus, it assists hospital management
in evaluating the costs and benefits of
implementing alert systems It also assists the
developers of alert systems in evaluating system
impacts and thus in improving the system and
justifying its widespread use
4.1 Evaluating the costs of medical alert systems
Figure 1 represents the framework for evaluating the costs associated with developing, implementing, using and maintaining an alert system It is assumed that these processes use four different types of resources: 1) IS personnel time, 2) hospital personnel time, 3) infrastructure and 4) capital
In the case of medical alert systems, the IS personnel cost and the hospital personnel cost are fairly easy to estimate The development and maintenance of software is usually done by external medical software vendors The associated cost is the amount that a hospital pays for the software vendor as software licenses and maintenance fees
In the future, some hospitals specialise in maintaing knowledge databases in specific areas Other hospitals can then purchase new versions of knowledge database against predefined fee Thus, the hospital personnel costs as well can be easily evaluated It is important to note, however, that the implementation and use of an alert system in a hospital is likely to require some time of physicians Obviously, the time of physicians should not be considered as a free resource
Capital employed in IS/IT
IS personnel time
Hospital personnel time
Hardware;
supplies;
office space
The development and maintenance
of alert systems
RESOURCE
IS personnel wages, vendor payments
Hospital personnel wages
Overhead costs
Interest costs COSTS
Figure 1: Overall costs of developing and maintaining alert information systems
Trang 5Electronic databases are a prerequisite for the use
of alert systems Investments needed in developing
and maintaing electronic patient databases,
electronic laboratory test databases, etc far exceed
the direct costs of developing alert systems These
costs are infrastructural in the sense that there are
many other applications that use the databases The
way these infrastructural costs should be taken into
account in evaluation is likely vary from one
hospital to another Finally, if the initial investment
is significant, the cost of capital employed in the
project should also be considered
4.2 Evaluating the benefits of medical
alert systems
Figure 2 represents the model for analysing the
impact of alert system on hospital operations The
model assumes that an alert information system can
reduce the time of physician in analysing potential
interactions Alternatively, it can improve the
quality of interaction information, which leads to
better clinical decisions and thus to cost savings or
improved health outcomes Finally, statistics about
the frequency and impacts of different alerts can
result to individual and organizational learning
In the absence of alert systems, physicians use their personal experience and judgement in evaluating the possible drug and medication inferences They consult colleagues, laboratory physicians, books and articles to consider the potential inferences while making clinical decisions Since an alert may reduce the need for such consultation, it has the potential to save physician time in making clinical decisions
The main objective in implementing alert systems
is to improve the quality of clinical decisions The evaluation should demonstrate that the implementation of an alert system has resulted to a permanent change in clinical decision making It should also demonstrate a change in the use of laboratory analyses and medication when diagnosing or treating a particular illness
The evaluation of whether the alert system impacts the quality of clinical decisions can be made in a number of ways Subjective assessment of physicians is one alternative Another approach is
to use laboratory tests using two groups of physicians to solve a number of clinical decision problems, one group using the alert system while the other one relying on their own judgement Ultimately, however, the evaluation should be based on the analysis of hospital records about actual clinical decisions
Improved evaluation of treatment
No information impact
Improved medical information
Resource savings in processing medical data
Resource savings in operations Alert
information
system
INFORMATION IMPACT
ORGANISATIONAL IMPACT
BENEFIT
Cost savings (information processing)
Cost savings
in hospital operations
Health outcomes/ patient utility
Individual/ organizational learning
Figure 2: Evaluation of alert/reminder system impacts
Improved statistics
DECISION IMPACT
Improved diagnosis / treatment decisions Improved
treatment
Trang 7Reduced need for laboratory experiments or
medication can lead to cost savings and/or improved
health outcomes In general, cost savings can be
expressed in monetary terms Most hospitals already
have cost estimates for a laboratory test, a particular
medication or a patient day in the hospital In
evaluating the impact on health outcomes or utility
for patients, subjective evaluation of the physicians
is perhaps most common Methods from health
economics could, however, be applied to get more
objective evaluation measures
Finally, it is possible that alert systems can constitute
an effective means to foster individual and
organizational learning about different interactions
Thus, physicians may learn about new interactions
and their influence on clinical decisions In the long
run, this may be an important outcome of
implementing an alert system
5 CANDELA information system
CANDELA is an information system that
automatically gives alerts of important drug effects
and drug interactions (other than therapeutic) on
laboratory tests It plays an active role in assisting
physicians in interpreting laboratory analysis results
and has potential both to reduce costs and to improve
the quality of health care services
CANDELA is based on a database that contains a
large number of rules about how different drugs
interact on laboratory tests The system is connected
to an electronic patient database that contains
information about the medication of individual
patients Thus, for each laboratory test the system
automatically checks whether patients medication
profile could interfere with the test results The ward
physician responsible for the treatment immediately
evaluates on-line alarms of potential drug
interactions (Grönroos et al 1995 a; Grönroos et al.
1995 b; Grönroos et al 1997).
In its current form, CANDELA generates alerts that
are automatically printed to laboratory test reports
Hence, the physicians are not direct users of the
system, and they can not directly ask for
explanations about the alerts If a physician wants
further information, he or she can use the system
interactively from a terminal
In general, CANDELA represents a fairly new type
of clinical information system A somewhat similar
system (HELP) has been in use in a hospital in Salt
Lake City, Utah User attitudes towards HELP have
also been evaluated (Gardner & Lundsgaarde 1994)
As more and more information about patients, drugs,
laboratory tests etc is converted to electronic form,
developing alert and reminder systems will become
easier Hence, the ability to evaluate such systems
becomes increasingly important
6 Evaluation plan for CANDELA information system
The CANDELA information system will be implemented in Turku University Central Hospital (TUCH) during the Fall 1997 The evaluation of CANDELA is projected to take place during Fall
1997 and Spring 1998 The objective of evaluation is
to identify and demonstrate the cost-effectiveness of CANDELA This is considered as an important prerequisite for its widespread use in other hospitals Evaluation is also expected facilitate systems implementation Because of the diversity of alerts, it seems necessary that the evaluation process leads to
a clear view about which of the CANDELA alerts are most valuable By doing so, evaluation assists in withholding some of the less valuable alerts from laboratory reports and thus in reducing information overflow
Based on the framework presented in figure 2, the essential questions for CANDELA evaluation can now be stated as follows:
1 does CANDELA improve the quality of information about drug interactions on laboratory tests?
2 does CANDELA have an impact on clinical decisions that are based on laboratory tests?
3 does the benefits of using CANDELA exceed the costs associated with it?
Hence, the objective is to evaluate the CANDELA system on multiple levels In the following, a plan for the evaluation is outlined
6.1 User satisfaction
The impact of CANDELA on information quality will be evaluated using Doll and Torkzadeh's (1988) UIS instrument This particular instrument was selected because its reliability has been tested in previous studies (Torkzadeh & Doll 1991;
Hendrickson et al 1994; Torkzadeh et al 1994) The
instrument contains key questions of user satisfaction
on information and its quality The developers of the instrument consider it as a potential surrogate measure for utility in decision-making (Doll & Torkzadeh 1988)
The questionnaire will be sent to physicians few months after the system has been taken to use The analysis aims at identifying the departments that were most satisfied with the information The impact
of variables such as age, job experience, specialisation, etc on the perceived value of alert information is analysed
Trang 86.2 Invidual impact
The impact of CANDELA on clinical decisions will
be analysed using two methods To get qualitative
data about the impact of alerts, some physicians are
asked to re-evaluate their previous clinical decisions
(that they had made prior systems implementation)
The question is, whether their decision would have
been different if CANDELA alerts had been
available at the time they made the decision
The second analysis relies on perceptions of
physicians as they use the alerts for interpreting
laboratory tests For each alert, physicians are asked
to evaluate the degree to which the alert was relevant
and the degree to which it changed their clinical
decision This analysis assists in identifying the
alerts that are most influential in decision making To
improve the validity of analysis, the results
concerning different alerts are later evaluated and
discussed with physicians, who are considered as
best experts in their field
6.3 Organizational outcome
The evaluation of organizational outcomes is based
on the analysis of historical data about clinical
decisions The hospital has history records about
laboratory tests, patients’ medical profiles and
associated clinical decisions from the past two years
The analysis of this data can reveal how frequently
physicians have misinterpreted laboratory tests
Hence, it assists in estimating the potential impact
that CANDELA can have on clinical decisions
The analysis of improved health outcomes relies
largely on the expert opinion of physicians Some of
the drug interactions on laboratory results may have
lead to inconclusive diagnosis and to further
experiments and medication Unnecessary
experiments or medication is likely influence
patients’ health The value of avoiding these tests can
be truly significant for an individual patient The
evaluation of health impacts is based on expert
opinion of physicians Health economics evaluation
methods will be applied as well
Evaluating the costs of CANDELA is primarily
based on the license prices and maintenance fees for
the software and for the drug interaction database
Subjective evaluation of the physicians will be used
to estimate whether the use of the system has
increased the time they need for analysing laboratory
results and making clinical decisions The
development costs of electronic patient database and
electronic laboratory database are considered as
infrastructural It seems difficult to assign any
monetary value for the use of the databases
In general, the evaluation of CANDELA is based on
historical comparisons It is believed that the
implementation of CANDELA will reduce the frequency of inaccurate clinical decisions in certain medical areas In this study, the primary objective is
to validate these impacts through history data Comparisons with other hospitals are not planned to
be made in the first round of evaluation
CANDELA information system is telling to physicians responsible for treatment what is true about patient What is true about patient, can theoretically be made without consideration of cost and benefit (comp Shortliffe 1987 p 62-64) But it would be more useful if information from cost and effects would be in future build into Computer Assisted Notification of Drug Effects on Laboratory Tests information system
7 Conclusions and future research
The framework for evaluating medical alert systems benefits researchers Its contribution is as a new means to help them study the impacts of such systems in clinical decision making The framework presented here can also be used in explaining why some alert systems provide more value in terms of cost savings and health outcomes than others do Practitioners can use the framework to review the potential benefits of alert systems in their own hospital They can establish objectives for implementing such systems And they can use the framework as an investigative tool in analysing reasons for low organizational impact of an implemented alert system
This paper describes the evaluation plan for CANDELA Its systematic evaluation will start simultaneously The first preliminary evaluation results will be available in October 1997 The full evaluation of the system is to be completed by May
1998 Based on the experiences gained in CANDELA evaluation the framework and methods will be improved and then applied in other medical information systems projects
Acknowledgements
This study is part of larger research program "Road ahead in Medical Informatics" We would like to express our gratitude to the Academy of Finland for their financial support to this program and to our research
References
Bailey J E (1990) Development of an Instrument for the Management of Computer User Attitudes in
Hospitals Methods of Information in Medicine.
1990:1, 51-56
Trang 9Bailey J.E - Pearson S.W (1983) Development of a
tool for measuring and analyzing computer user
satisfaction Management Science 1983:5, 530-545.
Chin, H.L - McClure, P (1995) Evaluating a
Comprehensive Outpatient Clinical Information
System: A Case Study and Model for System
Evaluation Proceedings Nighteenth Annual
Symposium on Computer Applications in Medical
Care JAMIA 1995 Lousiana.
DeLone, W.H - McLean, E.R (1992) Information
Systems Success: The Quest for the Dependent
Variable Information Systems Research 1992:1,
60-95
Dickson, G - Chervany N.- Senn J (1977) Research
in MIS: The Minnesota Experiments Management
Science 1977:5, 913-923.
Doll, W.J - Torkzadeh, G (1988) The Measurement
of End-User Computing Satisfaction MIS quarterly.
1988:2, 259-273
Drummond, M.F - Stoddart, G.L - Torrance, G.W
(1987) Methods for the Economic Evaluation of
Health Care Programmes Oxford University Press:
Oxford
Dupuits, F.M.H.M - Hasman, A (1995) User
satisfaction of general practioners with HIOS+, a
medical decision support system Computer Methods
and Programs in Biomedicine 1995:2, 183-188.
Galetta, D.F - Lederer, A.L (1989) Some Cautions
on the Measurement of User Information
Satisfaction Decision Sciences 1989:3, 419-439.
Gardner, R.M - Lundsgaarde, H.P (1994)
Evaluation of User acceptance of a Clinical Expert
System JAMIA 1994:6, 428-438.
Grönroos, P - Irjala, K - Forström, J.J (1995a)
Coding Drug Effets on Laboratory tests for Health
Care Information Systems JAMIA Symposium
Supplement, SCAMC Proceedings 1995, 449-453.
Philadelphia
Grönroos, P - Irjala, K - Heiskanen, J - Torniainen,
K - Forström J.J (1995b) Using computerized
individiual medication data to detect drug effects on
clinical laboratory tests Scandinavian journal of
clinical & laboratory investigation 1995:55, 31-36
(Suppl 222)
Grönroos, P.E Irjala, K.M Huupponen, R.K
-Scheinin, H - Forström, J - Forström, J.J (1997) A
Medication Database - A Tool for Detecting Drug
Interactions in Hospital European Journal of
Clinical Pharmalogy (in press)
Grover, V - Jeong, S.R - Segars, A.H (1996) Information systems effectiveness: The construct
space and patterns of application Information &
Management 1996:4, 117-191.
Hendrickson, A.R - Glorfeld, K - Cronan T.P (1994) On the Repated Test-Retest Reliability of the End-User Computing Satisfaction Instrument: A
Comment Decision Science 1994:4, 655-665.
Ingasol, G L - Hoffart, N -Schultz, A W (1990) Health services research in nursing: current status
and future directions Nursing Economics 1990:8,
229-238
Ives, B - Olson, M.H - Baroudi, J.J (1983) The Measurement of User Information Satisfaction
Communications of the ACM 1983:10, 785-793.
Johnston, R.H - Vitale, M.R (1988) Creating Competitive Advantage With Interorganizational
Information Systems MIS Quarterly 1988:2,
153-165
van der Loo, R.P - van Gennip, E.M.S.J - Baker, A.R (1995) Evaluation of automated information systems in health care: an approach to classifying
evaluative studies Computer Methods and Programs
in Biomedicine 1995:48, 45-52.
Maria B.L Lambay, F.A Dankel II, D -Chakravarthy, S - Tufekci S - Marcus, R -Kedar,
A (1994) XNEOr: Development and Evaluation of
an Expert System to Improve the quality and Cost of
Decision-Making in Neuro-Oncology Proceedings
Eighteenth Annual Symposium on Computer Applications in Medical Care JAMIA 1994,
678-683 Washington, DC
Mason, R.O (1978) Measuring Information Output:
A Communication System Approach Information &
Management 1978:5, 219234.
Matlin, G (1979) What Is the Value of Investment in
Information Systems? MIS Quarterly 1979:3, 5-34.
Menon, N.M - Lee, B - Eldenberg, L (1996) Information Technology Productivity in the Health
Care Industry Proceedings of he Seventeenth
International Conference on Information Systems.
1996, 477 Cleveland.
Miller J (1989) Information systems effectiveness: The fit between business Needs and system
capabilities Proceedings of the Tenth Internatinal
Conference on Information Systems 1989, 273-288 Boston Massachusetts (edirors DeGross J.I.,
Henderson J.C and Konsynski B.R.) Mitra, S - Karim, C.A (1996) Analyzing Cost-effectiveness of organizations: The Impact of
Trang 10Information Technology Spending Journal of
Management Information Systems 1996:2, 29-57.
Money A - Tromp D - Wegner T (1988) The Quantification of Decision Support Benefits Within
The Context of Value Analysis MIS Quarterly
1988:2, 223-236
O’Keefe, R.M (1989) The Evaluation of Decision-Aiding Systems: Guidelines and Methods
Information & Management 1989:17, 217-226.
Ovid MEDILINE (1966-1997)
Parker, M.M - Benson, R.J - Trainor, H.E (1988) Information ecomics Prentice-Hall Inc: New Jersey Pugh G E - Tan J K H (1994) Conputerized Databases for Emergency Care: What Impact on
Patient Care? Methods of Information in Medicine.
1994:5, 503-517
Saarinen T (1993) Success of information systems -Evaluation of Development Projects and the Choice
of Procurement and Implementation Strategies Acta Academiae Oeconomicae Helsingiensis Dissertation A:88:1993 Helskinki
Sanders, L.G - Courtney, J.F (1985) A Field Study
of Organizatinal Factors Influencing DSS Success
MIS Quarterly.1985:1, 77-93.
Scott, J.E (1994) The Measurement of Information Systems Effectiveness: Evaluating A Measuring
Instrument Proceedings of the 15th ICIS 111-128.
Vancouver
Shortliffe E.H (1987) Computer Programs to
Support Clinical Decision Making JAMA 1987:1,
61-66
Torkzadeh, G - Doll, W.J (1991) Test-retest reliability of the end-user computing satisfaction
instrument Decision Sciences 1991:1, 26-37.
Torkzadeh, G - Xia, W - Doll, W.J (1994) A Confirmatory Factor Analysis of the End-User
Computing Satisfaction Instrument MIS Quarterly.
1994:4, 453-461
Wagner, M.M - Cooper, G.F (1995) Evaluation of a Belief-Network-Based Reminder System that Learns
from Utility Feedback Proceedings of the
Nineteenth Annual Symposium on Computer Applications in Medical Care JAMIA 1995,
666-672 Lousiana
Zielstroff R.D (1985) Cost Effectiveness of Computerization in Nursing Practice and Administration The Journal of Nursing Administration 1985:2, 22-26.