close

Вход

Забыли?

вход по аккаунту

JP2017085372

код для вставкиСкачать
Patent Translate
Powered by EPO and Google
Notice
This translation is machine-generated. It cannot be guaranteed that it is intelligible, accurate,
complete, reliable or fit for specific purposes. Critical decisions, such as commercially relevant or
financial decisions, should not be based on machine-translation output.
DESCRIPTION JP2017085372
Abstract: A communication system, a communication apparatus, a communication method and a
program capable of performing intended output are provided. A communication apparatus
according to the present invention includes an area division unit, a transmission control unit, and
an output control unit. The area division unit at least includes a captured image obtained by
imaging the periphery of another communication device, and a display area indicating an area in
which the captured image is displayed in the output video output from the own device, It is
divided into a plurality of unit areas according to the number of persons reflected in the captured
image. The transmission control unit performs control to transmit coordinate information
indicating a unit area corresponding to the position pointed by the user in the output video to
another communication device. The output control unit performs control to output the received
output information when the output information including the sound whose directivity is
controlled according to the area corresponding to the coordinate information in the captured
image is received from the other communication device. . [Selected figure] Figure 5
Communication system, communication apparatus, communication method and program
[0001]
The present invention relates to a communication system, a communication apparatus, a
communication method, and a program.
[0002]
As one form of a communication system for transmitting and receiving data between a plurality
09-05-2019
1
of communication devices, a video (television) conference system that realizes a remote
conference using a communication network is known.
In this video conference system, in a conference room where one party such as an attendee who
performs remote conference is present, an image and a speech etc of a conference room such as
a conference party using a communication device (terminal device) of the remote video
conference system Voice is converted to digital data and transmitted to the other party's
communication device, the image is displayed on the display of the other party's conference
room, and the voice is output from the speaker, so that the remote conference can be performed
in a state close to an actual conference It can be performed.
[0003]
In such video conferencing systems, microphones are used to capture the audio of conference
participants, and cameras are used to capture video. Because the camera has an angle of view, it
is not possible to capture images of conference participants outside the angle of view of the
camera. In order to solve this problem, a method is known that uses a panoramic camera capable
of acquiring a 360-degree panoramic image. On the other hand, because ordinary microphones
are nondirectional, they will also collect surrounding sounds other than the participants'
comments. In order to solve this problem, it is known to use a microphone array to give
directivity to the sound collection range of the microphone, thereby suppressing the sound
collection of surrounding sounds and clearing the participants' speech. For example, Patent
Document 1 discloses a method in which a conference terminal forms a sound collection beam of
a microphone array in accordance with an arrangement pattern of a selected participant. Further,
Patent Document 2 discloses a technique of detecting the direction of a speaker using a
microphone array configured by arranging a plurality of microphones, and tracking the shooting
direction of the camera in that direction.
[0004]
A combination of the above panoramic camera and a microphone array is used to shoot the
entire conference room with the panoramic camera, and when there is a speaker, the microphone
sound collection range is directed at least in the direction of the speaker to make it more
realistic. There is a way to realize video conferencing with However, in this combination,
although the microphone sound collection range can be directed to the direction of the speaker,
the communication apparatus on the other side may not want the speaker to have an output that
09-05-2019
2
is focused. Therefore, in this combination, there is a problem that each communication apparatus
which constitutes a communication system can not perform an intended output.
[0005]
The present invention has been made in view of the above, and it is an object of the present
invention to provide a communication system, a communication device, a communication
method, and a program capable of performing intended output.
[0006]
In order to solve the problems described above and achieve the object, the present invention
provides a communication that includes a first communication device and a second
communication device that transmits and receives data between the first communication device.
In the system, the first communication device includes at least a captured image obtained by
imaging the periphery of the second communication device, and an output image showing an
image output by the first communication device. Among them, a display area indicating the area
in which the captured image is displayed is divided into a plurality of unit areas according to the
number of persons reflected in the captured image, and the user points out by the user in the
output video A first transmission control unit that performs control to transmit coordinate
information indicating the unit area corresponding to the specified position to the second
communication device; and pointing according to an area corresponding to the coordinate
information in the captured image Outputs including gender controlled voice An output control
unit that performs control to output the received output information when information is
received from the second communication device, and the second communication device acquires
the captured image; and When the coordinate information is received from the first
communication device, the captured image acquired by the acquisition unit based on
correspondence information indicating a correspondence between the coordinates of the output
video and the coordinates of the captured image. Among the plurality of microphones mounted
on the second communication device, directivity among the plurality of microphones mounted on
the second communication device is specified for the microphone corresponding to the region
specified by the specifying unit. A directivity control unit for controlling directivity of a
microphone array including the plurality of microphones to direct the first communication, and
the output information including at least voice whose directivity is controlled by the directivity
control unit; Comprising a second transmission control unit that controls to be transmitted to the
location, the.
[0007]
According to the present invention, it is possible to provide a communication system, a
communication apparatus, a communication method and a program capable of performing
09-05-2019
3
intended output.
[0008]
FIG. 1 is a diagram showing an example of the configuration of a communication system.
FIG. 2 is a diagram illustrating an example of a hardware configuration of the conference
terminal.
FIG. 3 is a diagram showing an example of installation of a microphone.
FIG. 4 is a diagram for explaining the details of the conference terminal. FIG. 5 is a diagram
showing an example of the hardware configuration of the projector. FIG. 6 is a schematic view
showing an example of the state of the video conference in the embodiment. FIG. 7 is a diagram
for describing a method of designating a cutout range of a projection image. FIG. 8 is a diagram
for explaining the cutout image. FIG. 9 is a diagram illustrating an example of the function of the
projector. FIG. 10 is a view showing an example of a projection image. FIG. 11 is a diagram
illustrating an example of functions of the control unit of the conference terminal. FIG. 12 is a
flowchart showing an operation example of the projector. FIG. 13 is a flowchart showing an
operation example of the conference terminal. FIG. 14 is a flowchart showing an operation
example of the conference terminal. FIG. 15 is a flowchart showing an operation example of the
conference terminal. FIG. 16 is a view showing an example of a projection image. FIG. 17 is a
diagram for explaining the method of dividing the display area according to the modification.
[0009]
Hereinafter, embodiments of a communication system, a communication device, a
communication method, and a program according to the present invention will be described in
detail with reference to the attached drawings.
[0010]
FIG. 1 is a diagram showing an example of the configuration of the communication system 1 of
the present embodiment.
09-05-2019
4
In the example of FIG. 1, one conference terminal 2 is disposed at each of the base A and the
base B. The conference terminals 2 disposed at each of the base A and the base B are connected
to the server 4 via the network 3 such as the Internet. The present invention is not limited to this,
and the number of conference terminals 2 (the number of bases) included in the communication
system 1 can be arbitrarily changed.
[0011]
The server 4 monitors whether or not each conference terminal 2 is connected to the server 4,
and performs necessary control such as call control of the conference terminal 2 at the start of
the conference. At the time of a conference, the conference terminal 2 transmits image and audio
data to the server 4 at the time of data transmission of its own device, and the server 4 transmits
image and audio data to another conference terminal 2 on the other side. Do. At the time of data
reception, image / voice data of the other conference terminal 2 on the opposite side is received
via the server 4. For example, when a meeting is held at the base A and the base B, the data
transmitted by the conference terminal 2 at the base A is transmitted to the conference terminal
2 at the base B via the server 4 and another conference terminal 2 Not sent to the conference
terminal 2) which is not Similarly, the data transmitted by the conference terminal 2 at the site B
is transmitted to the conference terminal 2 at the site A participating in the conference via the
server 4 and to the other conference terminals 2 not participating in the conference. Not sent By
performing the control as described above, it is possible to conduct a conference between a
plurality of conference terminals 2 (a plurality of bases).
[0012]
Next, the configuration of the conference terminal 2 will be described. In addition, since the
structure of the meeting terminal 2 arrange ¦ positioned at each of the base A and the base B is
the same, in the following description, the one meeting terminal 2 is mentioned as an example,
and is demonstrated. FIG. 2 is a diagram showing an example of the hardware configuration of
the conference terminal 2. As shown in FIG. 2, the conference terminal 2 includes a panoramic
camera 10, a display unit 11, a microphone array 12, a speaker 13, a CPU 14, a storage device
15, a memory 16, a LAN I / F unit 17, and an operation unit 18.
[0013]
09-05-2019
5
The panoramic camera 10 is an example of an imaging unit , and generates a captured image
obtained by imaging. In this example, the panoramic camera 10 generates a panoramic image (an
example of a captured image) obtained by imaging 360 degrees around the panoramic camera
10 (which may be considered to be 360 degrees around the conference terminal 2), The
generated panoramic image is transmitted to the CPU 14. For example, the panoramic camera 10
is configured of a known omnidirectional camera or the like. By imaging 360 degrees around the
panoramic camera 10, it is possible to generate a captured image in which all the conference
participants present around the panoramic camera 10 are reflected. Here, a panoramic image is
an image generated by combining a plurality of images (captured images) by shooting with a
camera having a plurality of imaging elements, moving one camera, or using a plurality of
cameras. Means In this example, the area where imaging by the panoramic camera 10 is
performed is 360 degrees around the panoramic camera 10, but may be narrower than this.
[0014]
The display unit 11 has a function of displaying the video data received from the CPU 14 on the
screen. In this example, the display unit 11 is configured of a liquid crystal display device or the
like.
[0015]
The microphone array 12 includes a plurality of microphones distributedly installed in the
conference terminal 2 and has a function of acquiring voices of conference participants and
transmitting the voices to the CPU 14. The microphone array is composed of a plurality of
omnidirectional microphones (microphones) and forms directivity by a beam former. In general,
the beam former is a method of forming directivity by utilizing the time difference of the sound
reaching each microphone. The directivity formed by the beam former can be freely directed in
the vertical and horizontal directions by the arrangement of the microphones. Therefore, it is
possible to arbitrarily change the area (the target area) where sound collection is desired.
[0016]
FIG. 3 is a diagram showing an example in which six microphones a to f included in the
09-05-2019
6
microphone array 12 are installed in the case of the conference terminal 2. The CPU 14 can
collect voices in an arbitrary range by switching on / off of the respective microphones and
integrating the voices of the respective microphones. Further, the correspondence between each
microphone and the installation location in the case of the conference terminal 2 is stored in
advance in the storage device 15 of the conference terminal 2.
[0017]
The description of FIG. 2 is continued. The speaker 13 has a function of outputting audio data
received from the CPU 14. The CPU 14 centrally controls the operation of the entire conference
terminal 2. For example, the CPU 14 has a function to control a video conference, encodes video
acquired from the panoramic camera 10 and audio data acquired from the microphone array 12
and transmits it to the LAN I / F unit 17 and receives it by the LAN I / F unit 17 It has a CODEC
function or the like that decodes video and audio data of the other party of the conference and
transmits it to the display unit 11 and the speaker 13. As an example of the CODEC used by the
CPU 14, H. 2. H.264 / AVC and H.264. There are H.264 / SVC. Furthermore, the CPU 14 has a
function of controlling the directivity of the microphone array 12, a function of close-up and
displaying the speaker among the conference participants reflected in the panoramic image
acquired from the panoramic camera 10, and the like.
[0018]
The storage device 15 stores various control programs (video conference control and the like)
executed by the CPU 14, a conversion table described later, and the like. Examples of the storage
device 15 include non-volatile storage media such as flash memory and HDD.
[0019]
The memory 16 develops a program to be executed by the CPU 14 and temporarily stores
operation data. Examples of the memory 16 include volatile memory such as DDR memory. The
LAN I / F unit 17 is connected to another conference terminal 2 via the network 3 to transmit
and receive data (image / voice data). The LAN I / F unit 17 supports 10Base-T, 100Base-TX, and
1000Base-T, and is a wired LAN connected to Ethernet (registered trademark), and a wireless
LAN compatible with 802.11a / b / g / n / ac. Are mentioned as an example.
09-05-2019
7
[0020]
The operation unit 18 is a device used for various operations by the user (various operations
related to device control of the conference terminal 2), and includes, for example, a keyboard,
buttons, and the like.
[0021]
FIG. 4 is a diagram for explaining the details of the conference terminal 2.
The conference terminal 2 is connected to a projector 300 which is an example of an output
device. The projector 300 has a function of projecting an image input from the conference
terminal 2 on a projection surface (for example, a screen) provided at a base where the projector
300 is installed. The projector 300 performs various operations (point operation, click operation,
scroll operation) by directly operating the area where the image is projected on the projection
surface using a special wireless interactive pen (special stylus pen). So-called interactive
operation is possible, for example. With TI's DLP interactive projector, a system that can be
integrated into the projection beam and managed at its own pixel level allows the projector to
keep track of where on the projection plane a dedicated stylus is pointing There is a feature that
it is not necessary to execute calibration at startup and calibration is not necessary even when
the projector is moved. In the present embodiment, a projector 300 compatible with this system
and a dedicated stylus pen 400 are used.
[0022]
The projector 300 has a function of wirelessly communicating with each of the stylus pen 400
and the conference terminal 2, and the stylus pen 400 has a function of wirelessly
communicating with the projector 300. Since the projector 300 can acquire information
indicating the position pointed to by the stylus pen 400 on the projection surface from the stylus
pen 400 by wireless communication with the stylus pen 400, the position pointed to by the
stylus pen 400 on the projection surface (by the user It can always be understood that it may be
considered to be the pointed position. Also, the wireless communication with the projector 300
enables the conference terminal 2 to always grasp the position indicated by the stylus pen 400
on the projection surface.
09-05-2019
8
[0023]
In this example, one combination of the projector 300 having the same configuration and the
dedicated stylus pen 400 is disposed at each of the base A and the base B. Here, the conference
terminal 2 and the projector 300 connected to the conference terminal 2 correspond to the
first communication device or the second communication device . In addition, the kind of
output device connected to the meeting terminal 2 is arbitrary, for example, the form in which an
interactive whiteboard is connected to the meeting terminal 2 may be sufficient.
[0024]
Further, for example, a single apparatus having the function of the conference terminal 2 and the
function of the projector 300 may be installed at each site. In this form, the single device
corresponds to the "first communication device" or the "second communication device".
[0025]
As shown in FIG. 4, the conference terminal 2 includes the network unit 100, the control unit
101, the display control unit 102, the wireless communication unit 109, in addition to the
panoramic camera 10, the microphone array 12, the speaker 13, and the storage device 15
described above. A camera I / F receiver 111, a lens characteristic holding unit 112, and a
distortion correction processing unit 113 are provided. For each function of the network unit
100, the control unit 101, the display control unit 102, the wireless communication unit 109, the
camera I / F receiver 111, and the distortion correction processing unit 113, the CPU 14 stores
the program stored in the storage device 15 or the like. It may be in a form realized by execution,
or may be a form in which at least a part of these functions is realized by a dedicated hardware
circuit (such as a semiconductor integrated circuit). Also, for example, the lens characteristic
holding unit 112 may be realized by the storage device 15.
[0026]
The network unit 100 transmits and receives data to and from another conference terminal 2
which is the other party of the conference.
[0027]
09-05-2019
9
The control unit 101 is a unit that performs various controls and calculations.
Details of the functions of the control unit 101 will be described later.
[0028]
The display control unit 102 has a function of controlling display (projection of an image on a
projection surface) by the projector 300.
[0029]
The wireless communication unit 109 performs wireless communication with the projector 300,
and acquires from the projector 300 position information indicating a position indicated by the
stylus pen 400 on the projection surface.
The wireless communication unit 109 can notify the control unit 101 of the position information
acquired from the projector 300.
[0030]
The panoramic image generated by the panoramic camera 10 is sent to the camera I / F receiver
111. The camera I / F receiver 111 is assumed to be a high-speed serial I / F such as VbyOne
(registered trademark) or HDMI (registered trademark). The distortion correction processing unit
113 corrects the distortion of the panoramic image that has been subjected to serial / parallel
conversion by the camera I / F receiver 111, and is output to the control unit 101. The lens
characteristic storage unit 112 stores conversion parameters for performing distortion
correction in accordance with the lens characteristics, and the distortion correction processing
unit 113 can correct distortion of a panoramic image using the conversion parameters.
[0031]
09-05-2019
10
FIG. 5 is a schematic diagram showing an example of the hardware configuration of the projector
300. As shown in FIG. As shown in FIG. 5, the projector 30 includes a CPU 311, a storage unit
312, an input unit 313, a communication I / F 314, and a projection unit 315, which are
connected via a bus.
[0032]
The CPU 311 executes a program stored in the storage unit 312 to centrally control the
operation of the projector 300. The storage unit 312 is configured by a ROM or an HDD that
stores a program executed by the CPU 311 or data required to execute the program, and a RAM
that functions as a work area of the CPU 311 or the like. The input unit 313 is for performing
various inputs to the projector 300, and includes a touch panel, a key switch, and the like. The
communication I / F 314 is an interface for communicating with the stylus pen 400 and the
conference terminal 2. The projection unit 315 projects image data to be projected onto a
projection surface such as a screen. The projection unit 315 includes an optical system for
projection, such as a projection lens. The functions of the projector 300 will be described later.
[0033]
FIG. 6 is a schematic view showing an example of the state of the video conference according to
the present embodiment. A conference terminal 2 is placed at a desk at the base 1. Since the
conference terminal 2 is equipped with the panoramic camera 10, it is assumed that it is placed
at the center of the desk. As described above, the conference terminal 2 has the microphone
array 12 including the microphones a to f. Base 1 shows that five persons D to H participate in
the video conference. Further, the above-described projector 300 is connected to the conference
terminal 2 installed at the base 1 via the video output cable, and an image including a panoramic
image on the base 2 side (a panoramic image on the base 2 side is displayed) The image is
projected on the projection plane. In the following description, an image in a state of being
projected onto the projection plane on the base 1 side is referred to as projected image 1 ,
and an image in a state onto which the projection plane on the base 2 side is projected is referred
to as projection image 2 . When the two are not distinguished from one another, they are
simply referred to as projected images . Here, the projection image is an example of the
output image . Under the projection image 1, a panoramic image in which all persons
(persons A to C in this example) participating in the meeting at the base 2 are displayed is
displayed. Further, on the upper side of the projection image 1, a cutout image in which the
person A who is the speaker is enlarged is displayed. In the case of a video conference system
equipped with a normal speaker following function, when one of the conference participants
09-05-2019
11
speaks, the speaker is displayed in a close-up on the cutout image.
[0034]
The conference terminal 2 is also placed on the desk at the site 2. As in the case of the base 1,
since the conference terminal 2 is equipped with the panoramic camera 10, it is assumed that it
is placed at the center of the desk, and the microphone array 12 including the microphones a to f
is mounted. . Base 2 indicates that three persons A to C participate in the video conference.
Similar to the base 1, the above-described projector 300 is connected to the conference terminal
2 installed at the base 2 via a video output cable, and an image including a panoramic image on
the base 1 side (a panoramic image on the base 1 side) Is displayed on the projection plane.
Under the projection image 2, a panoramic image in which all persons (persons D to H in this
example) participating in the meeting at the base 1 are displayed is displayed. Here, normally, the
speaker is displayed in the cutout image, but as shown in FIG. 5, for example, any person
participating in the conference at the base 2 is surrounded by a dotted line using the stylus pen
400. When a specified area is specified, a voice whose directivity is controlled in accordance with
the specified area is output, and an image of the specified area is displayed on the projection
image 2 as a cutout image. That is, by designating an area where a person who wants to close up
is reflected among the conference participants reflected in the panoramic image, it is possible to
perform focused output in the designated area. In addition to the human being, by designating an
area in which a subject to be noticed is reflected among the subjects reflected in the panoramic
image, it is possible to perform focused output in the designated area. The configuration of the
conference terminal 2 for realizing this (the function of the control unit 101) will be described
later.
[0035]
A method of designating the cut-out range of the projection image will be described with
reference to FIG. When the projector 300 receives pressing of the start icon 310 for instructing
to start specification of the extraction range, the projector 300 displays a pop-up screen for
confirming start of specification of the extraction range as shown in FIG. Control is performed
(under the control of the display control unit 102). Then, when the projector 300 receives the
pressing of the OK button on the pop-up screen, the projector 300 performs control to
display a message prompting the user to specify the center point of cutout as shown in FIG. 7B.
Thereafter, the user can use the stylus pen 400 or the like to perform an operation of specifying
the center point of the cutout.
09-05-2019
12
[0036]
As will be described later, in the present embodiment, a plurality of display areas indicating an
area in which a panoramic image (a panoramic image on the other side of the conference) is
displayed among the projection images are provided according to the number of persons
reflected in the panoramic image. Are divided into unit areas, and the user points an operation
with the stylus pen 400 or the like as the central point of the cutout (not necessarily the central
point) of any person reflected in the display area. If it does, the unit area ¦ region containing the
central point of this cutting out will be designated as a cutting out range. As a result, since it is
possible to simplify the operation (the operation of specifying the clipping range) for specifying
the area in which the subject to be focused is reflected in the display area of the projection
image, the convenience of the user can be improved.
[0037]
Then, as shown in FIG. 7C, when the projector 300 receives the pressing of the end icon 320 for
instructing the end of the specification of the cut-out range, the projector 300 selects the
specified cut-out range of the projection plane ( In the example of FIG. 7, coordinate information
indicating coordinates of a rectangle enclosed by a dotted line is transmitted to the conference
terminal 2. Then, the conference terminal 2 performs control to transmit the range information
received from the projector 300 to another conference terminal 2 as a conference partner.
[0038]
Now, as shown in FIG. 8, the person H appears in the projection image 2 (it can also be regarded
as an area where an image including a panoramic image on the base 1 side is projected on the
projection surface on the base 2 side). It is assumed that the embedded area 330 is designated as
the cutout range. In this example, the area 330 is a rectangular area, and the coordinate
information of the area 330 is information indicating the coordinates of each of the four vertices
(A, B, C, D) of the area 330. In this example, the coordinates of vertex A in projection image 2 are
(Xa, Ya), the coordinates of vertex B are (Xb, Yb), the coordinates of vertex C are (Xc, Yc), and the
coordinates of vertex D are (Xd, Yd) ). This coordinate information is transmitted to the
conference terminal 2 on the base 1 side.
09-05-2019
13
[0039]
The conference terminal 2 on the base 1 side has a cutout image obtained by cutting out the area
corresponding to the coordinate information received from the conference terminal 2 on the
base 2 side among the panoramic images on the base 1 side, and the directivity is controlled
according to the area The output information including the generated voice is generated, and the
generated output information is transmitted to the conference terminal 2 on the base 2 side. In
the example of FIG. 8, the conference terminal 2 on the base 1 side is the coordinates (Xa ′, Ya
′) of A ′ and the coordinates (Ba) of the panoramic image acquired from the panoramic camera
10 mounted on the own device. An image of a rectangular area surrounded by Xb ′, Yb ′),
coordinates of C ′ (Xc ′, Yc ′), and coordinates of D ′ (Xd ′, Yd ′) is cut out as a cutout
image. The conference terminal 2 on the base 1 side is connected to the conference terminal 2
on the base 2 side among the panoramic images based on the position information indicating the
relationship between the position of the microphone included in the microphone array 12 and
the coordinates of the panoramic image. The directivity of the microphone array 12 is controlled
to direct the directivity to the microphone placed at the position closest to the coordinates of the
area corresponding to the received coordinate information.
[0040]
Then, the conference terminal 2 on the base 1 side transmits, to the conference terminal 2 on the
base 2 side, output information including the clipped image cut out as described above and the
voice whose directivity is controlled. The conference terminal 2 on the base 2 side outputs the
output information received from the conference terminal 2 on the base 1 side.
[0041]
Hereinafter, the detailed content of the function which each of the projector 300 and the control
part 101 of the conference terminal 2 has is demonstrated. First, the functions of the projector
300 will be described. FIG. 9 is a diagram illustrating an example of the function of the projector
300. As shown in FIG. 9, the projector 300 includes a projection control unit 321, an area
division unit 322, a unit area selection unit 323, and a coordinate information transmission
control unit 324. For convenience of explanation, FIG. 9 mainly illustrates the functions related to
the present embodiment, but the functions that the projector 300 has are not limited to these.
09-05-2019
14
[0042]
The projection control unit 321 performs control of projecting an image input from the
conference terminal 2 on a projection plane under the control of the display control unit 102.
[0043]
The area division unit 322 is a panoramic image obtained by imaging the periphery of another
conference terminal 2 as a conference partner among the projection images showing an image in
a state of being projected onto the projection plane by the own device (other conference terminal
2 side The display area showing the area in which the panoramic image of (1) is displayed is
divided into a plurality of unit areas according to the number of persons reflected in the
panoramic image.
In the present embodiment, the area dividing unit 322 may consider the number of the user's
operation to indicate the person reflected in the display area (the number of positions indicated
by the user's operation in the display area). A plurality of unit areas can be obtained by dividing
the display area at equal intervals according to.
[0044]
For example, it is assumed that the projection image output by the projector 300 is in the mode
of FIG. In the present embodiment, as in the case of the icon for specifying the cutout range
described above, pressing of the icon for instructing execution of calibration of the
correspondence between the designated position and the plurality of unit areas by the user on
the projection image Enters the calibration mode, and the area division unit 322 performs
control to output information (which may be video or audio) prompting the user to point to the
central points of a plurality of persons reflected in the display area. . In the example of FIG. 10,
since five persons D, E, F, G, and H are reflected in the display area, the user uses the stylus pen
400 or the like to point to the central point of those persons ( Perform the operation to instruct).
The number of points (the number of user operations for pointing a person) is the total number
of persons reflected in the display area, and in this example, the number of points is 5 .
Therefore, the area dividing unit 322 can obtain five unit areas (the unit area 401 to the unit
area 405) by equally dividing the display area into five in the horizontal direction (horizontal
direction). The calibration mode ends when the unit area is obtained.
09-05-2019
15
[0045]
The description of FIG. 9 is continued. The unit area selection unit 323 selects a unit area
corresponding to the position pointed by the user among the projection images output from the
own device. In this example, when the unit area selection unit 323 receives an operation of
specifying the above-described center point of cutout, the unit area selection unit 323 selects a
unit area including the coordinates of the specified center point of cutout.
[0046]
The coordinate information transmission control unit 324 performs control to transmit
coordinate information indicating the unit area selected by the unit area selection unit 323 to the
conference terminal 2 connected to the own apparatus.
[0047]
The functions of the projector 300 described above (the projection control unit 321, the area
dividing unit 322, the unit area selecting unit 323, and the coordinate information transmission
control unit 324) execute the program stored in the storage unit 312 or the like by the CPU 311.
Or at least a part of these functions may be realized by a dedicated hardware circuit (for example,
a semiconductor integrated circuit etc.).
[0048]
Next, functions of the control unit 101 of the conference terminal 2 will be described.
FIG. 11 is a diagram illustrating an example of functions of the control unit 101 of the
conference terminal 2.
For convenience of explanation, FIG. 11 mainly illustrates the functions related to the present
embodiment, but the functions that the control unit 101 has are not limited to these.
[0049]
As illustrated in FIG. 11, the control unit 101 includes a first transmission control unit 121, an
acquisition unit 122, a specifying unit 123, a cutout unit 124, a directivity control unit 125, and
09-05-2019
16
a second transmission control unit 126. And an output control unit 127.
[0050]
When the first transmission control unit 121 receives the above-described coordinate
information from the projector 300 connected to the own apparatus, the first transmission
control unit 121 performs control to transmit the received coordinate information to another
conference terminal 2 as a conference partner.
That is, the first transmission control unit 121 performs control to transmit coordinate
information indicating a unit area corresponding to the position pointed by the user in the
projection image to the other conference terminal 2 as the conference partner.
[0051]
The acquisition unit 122 acquires a panoramic image obtained by imaging with the panoramic
camera 10 that images the periphery of the own apparatus. In this example, the acquisition unit
122 acquires the panoramic image after correction input from the distortion correction
processing unit 113 described above.
[0052]
The identification unit 123 is a user of the other conference terminal 2 in the display area of the
projection image (the area on which the panoramic image of the own apparatus is displayed)
output from the other conference terminal 2 When coordinate information indicating a unit area
corresponding to the position pointed by is received, the coordinates of the projection image
(which may be considered as the coordinates of the area of the projection surface on which the
image is projected) and the coordinates of the panoramic image Among the panoramic images
acquired by the acquisition unit 122, an area corresponding to the received coordinate
information is specified based on the correspondence information indicating the correspondence.
In this example, the correspondence information is stored in advance in the storage device 15. In
addition, since a normal video conference system can display only the image of its own device,
display only the image of the other party, and can freely change the layout (change the display
mode), the coordinates of the projection image and the panoramic image The relationship with
09-05-2019
17
the coordinates of does not necessarily correspond one to one. Therefore, the correspondence
information in this example associates the coordinates of the projection image with the
coordinates of the panoramic image for each display mode (layout information) of the projector
300.
[0053]
The cutout unit 124 cuts out, as a cutout image, an image of the area specified by the specifying
unit 123 among the panoramic images acquired by the acquisition unit 122.
[0054]
The directivity control unit 125 directs directivity to the microphone corresponding to the area
(in this example, the area in the panoramic image) specified by the specifying unit 123 among
the plurality of microphones dispersedly mounted on the conference terminal 2. Control the
directivity of the microphone array 12.
The directivity control unit 125 determines a microphone corresponding to the coordinates of
the area specified by the specifying unit 123 based on position information indicating the
relationship between the position of the microphone included in the microphone array 12 and
the coordinates of the panoramic image. be able to. The position information may be stored, for
example, in the storage device 15 or the like.
[0055]
The second transmission control unit 126 performs control to transmit output information
including at least voice whose directivity is controlled by the directivity control unit 125 to
another conference terminal 2. In the present embodiment, the second transmission control unit
126 sends output information including the voice whose directivity is controlled by the
directivity control unit 125 and the cutout image cut out by the cutout unit 124 to another
conference terminal 2. Control sending. More specifically, the second transmission control unit
126 includes the panoramic image acquired by the acquisition unit 122, the sound whose
directivity is controlled by the directivity control unit 125, and the cutout image cut out by the
cutout unit 124. And control the transmission of the output information to the other conference
terminal 2. Note that the output information is a voice whose directivity is controlled by the
directivity control unit 125 (in accordance with the area corresponding to the coordinate
09-05-2019
18
information received from the other conference terminal 2 in the captured image acquired by the
acquisition unit 122) In any form as long as it contains at least a controlled voice). For example,
the cutout portion 124 may not be provided.
[0056]
When the second transmission control unit 126 has not received the above-mentioned
coordinate information from another conference terminal 2, the panoramic image acquired by
the acquiring unit 122 and the conference participant reflected in the panoramic image Control
is performed to transmit, to another conference terminal 2, normal conference information
including a clipped image obtained by slicing out the speaker and expanding and a voice data
collected by the microphone array 12.
[0057]
The output control unit 127 performs control to output an image and a sound received from
another conference terminal 2.
The output control unit 127 instructs the display control unit 102 to output (project on the
projection surface) an image received from the other conference terminal 2 from the projector
300, and the voice received from the other conference terminal 2 is output. Control to output
from the speaker 13 is performed. When the output control unit 127 of the present embodiment
receives the above-described output information from another conference terminal 2, the output
control unit 127 performs control to output the received output information. More specifically,
the output control unit 127 instructs the display control unit 102 to output an image obtained by
combining the cutout image and the panoramic image included in the received output
information, and the received output information Control of outputting the sound contained in
the speaker 13 from the speaker 13.
[0058]
Further, when the above-mentioned normal conference information is received from another
conference terminal 2, the output control unit 127 performs control to output the received
normal conference information.
[0059]
The functions of the control unit 101 described above (the first transmission control unit 121,
09-05-2019
19
the acquisition unit 122, the specifying unit 123, the cutout unit 124, the directivity control unit
125, the second transmission control unit 126, the output control unit 127) May be realized by
the CPU 14 executing a program stored in the storage device 15 or the like, or at least a part of
the functions possessed by the control unit 101 is a dedicated hardware circuit (for example, a
semiconductor integrated circuit etc.) May be realized by
[0060]
Moreover, although the panoramic camera 10 and the speaker 13 are contained in the meeting
terminal 2 in the above example, not only this but the form provided, for example out of the
meeting terminal 2 may be sufficient.
[0061]
FIG. 12 is a flowchart showing an example of the operation of the projector 300 in the case of
designating the clipping range.
When pressing of the start icon 310 is received (step S1: Yes), an operation for specifying the
center point of clipping is received (step S2).
Next, a unit area corresponding to the center point of the clipping specified in step S2 is selected
(step S3).
Then, when pressing of the end icon 320 is received (step S4: Yes), coordinate information
indicating the unit area selected in step S3 is transmitted to the conference terminal 2 (step S5).
[0062]
FIG. 13 is a flowchart showing an operation example of the conference terminal 2 when
coordinate information is received from the projector 300 connected to the apparatus itself.
When the coordinate information is received (step S6: Yes), the first transmission control unit
121 performs control to transmit the received coordinate information to the other conference
terminal 2 (step S7).
09-05-2019
20
[0063]
FIG. 14 is a flowchart showing an operation example of the conference terminal 2 when
coordinate information is received from another conference terminal 2. When coordinate
information is received from another conference terminal 2 (step S10: Yes), the identifying unit
123 acquires a panoramic image (obtained from the panoramic camera 10 of the own device)
acquired by the acquiring unit 122 based on the correspondence information described above.
An area corresponding to the received coordinate information is specified among the panoramic
images (step S11). Next, the cutout unit 124 cuts out, as a cutout image, an image of a region
corresponding to the region specified in step S11 among the panoramic images acquired by the
acquisition unit 122 (step S12). Next, the directivity control unit 125 sets the directivity of the
microphone array 12 to direct the directivity to the microphone corresponding to the area
specified in step S11 among the plurality of microphones dispersedly mounted on the conference
terminal 2. It controls (step S13). Then, the second transmission control unit 126 outputs output
information including the panoramic image acquired by the acquisition unit 122, the cutout
image cut out in step S12, and the sound whose directivity is controlled as a result of step S13.
Control to transmit to the conference terminal 2 is performed (step S14).
[0064]
FIG. 15 is a flowchart showing an operation example of the conference terminal 2 when output
information is received from another conference terminal 2. When the output information is
received from another conference terminal 2 (step S20: Yes), the output control unit 127
performs control to output the received output information (step S21).
[0065]
As described above, when the conference terminal 2 according to the present embodiment
receives the above-described coordinate information from another conference terminal 2 serving
as a conference partner, the conference terminal 2 receives a message from the panoramic
camera 10 of its own device based on the above-described correspondence information. Among
the acquired panoramic images, an area corresponding to the received coordinate information is
specified, and an image of the specified area is cut out as a cutout image. Further, the directivity
of the microphone array 12 is controlled to direct the directivity to the microphone
corresponding to the specified area among the plurality of microphones distributed and installed
09-05-2019
21
in the conference terminal 2. Then, the conference terminal 2 transmits output information
including the cutout image and the voice whose directivity is controlled to the other conference
terminal 2, and the other conference terminal 2 outputs the received output information.
Thereby, the other conference terminal 2 can perform the intended output. In the present
embodiment, as the panoramic camera, the range of the imaging region is 360 degrees around.
However, the gist of the present embodiment is to designate a part of the captured image as a
cutout image and control the directivity of the microphone corresponding to the cutout image.
Therefore, as an imaging region, the angle of view of the camera may be less than 360 degrees,
and may be, for example, about 80 degrees.
[0066]
As mentioned above, although embodiment concerning this invention was described, this
invention is not limited to the above-mentioned embodiment as it is, At an execution phase, a
component is changed and actualized in the range which does not deviate from the summary. In
addition, various inventions can be formed by appropriate combinations of a plurality of
components disclosed in the above-described embodiments. For example, some components may
be deleted from all the components shown in the above-described embodiment.
[0067]
In the above-described embodiment, the area dividing unit 322 obtains a plurality of unit areas
by dividing the display area at equal intervals according to the number of user operations for
pointing the person reflected in the display area. For example, as shown in FIG. 16, it is assumed
that the positions of a plurality of persons reflected in the display area are biased. In this case, if
the display area is equally divided in the horizontal direction by the number of persons reflected
in the display area, the plurality of unit areas obtained and the plurality of persons reflected in
the display area do not correspond one to one. As a result, it may be difficult to appropriately
designate (specify as a cutout range) an area in which a person to be focused is reflected.
[0068]
Therefore, for example, the area dividing unit 322 can set the display area to the display area
based on the relative positional relationship between the four vertices of the projection video and
the plurality of positions in the display area pointed to by the operation of the user. It is also
09-05-2019
22
possible to divide into a plurality of positions in the display area indicated by the operation and a
plurality of unit areas corresponding to one to one. Specific contents will be described below.
[0069]
When the above-described calibration mode is entered, the area division unit 322 performs
control to output information (which may be a video or an audio) prompting pointing to the four
vertices of the projection image. For example, when the output device connected to the
conference terminal 2 is an interactive whiteboard, this control is unnecessary because the
coordinates of four vertices of the image (an example of the output video) displayed by the
interactive whiteboard are known in advance. It becomes.
[0070]
Then, the area division unit 322 performs control to output information prompting pointing to
the center points of a plurality of persons reflected in the display area, and detects a plurality of
positions in the display area indicated by the user's operation. Do. By grasping the relative
positional relationship between a plurality of positions in the display area pointed to by the
user's operation (sometimes referred to as specified position in the following description)
and the four vertices of the display area It is possible to obtain the position of the cutout line (the
cutout line extending in the vertical direction) for cutting out a plurality of unit areas
corresponding to a plurality of designated positions and one-to-one.
[0071]
The following description will be given focusing on one designated position. Hereinafter, the
designated position focused on is referred to as a focused position . First, it is assumed that
there is a designated position adjacent to the left of the position of interest and a designated
position adjacent to the right. In this case, the horizontal position of the cutout line on the right
side for cutting out the unit area including the target position is, for example, the horizontal
length of the display area and the distance from the right side of the display area to the
horizontal position of the target position. The left side of the display area is a distance obtained
by adding a half of the distance between the lateral position of the position of interest and the
lateral position of the designated position adjacent to the right of the position of interest to the
difference of It can be determined as a position separated laterally from. Similarly, the horizontal
09-05-2019
23
position of the cutout line on the left side for cutting out the unit area including the position of
interest is, for example, the length in the horizontal direction of the display area and the distance
from the right side of the display area to the position in the horizontal direction of the target
position. From the left side of the display area by the distance obtained by further subtracting
1/2 of the distance between the lateral position of the position of interest and the lateral position
of the designated position adjacent to the left side of the position of interest from the difference
of It can be determined as a laterally separated position.
[0072]
Next, it is assumed that there is no designated position adjacent to the left of the focused position
and the designated position adjacent to the right. In this case, the lateral position of the right side
cutout line for cutting out the unit area including the target position can be obtained as in the
first case. On the other hand, the horizontal position of the cutout line on the left side for cutting
out the unit area including the position of interest is, for example, the horizontal length of the
display area and the distance from the right side of the display area to the position in the
horizontal direction of the target position. The difference between the horizontal position of the
position of interest and the horizontal position of the designated position adjacent to the right of
the position of interest further subtracts 1/2 of the distance from the left side of the display area.
It can be determined as a position separated in the direction.
[0073]
Furthermore, it is assumed that there is a designated position adjacent to the left of the focused
position and no designated position adjacent to the right. In this case, the lateral position of the
left cutout line for cutting out the unit area including the target position can be obtained as in the
first case. On the other hand, the horizontal position of the cutout line on the right side for
cutting out the unit area including the position of interest is, for example, the horizontal length of
the display area and the distance from the right side of the display area to the position in the
horizontal direction of the target position. From the left side of the display area, a distance
obtained by adding a half of the distance between the lateral position of the target position and
the lateral position of the designated position adjacent to the left of the target position to the
difference It can be determined as a laterally separated position.
[0074]
09-05-2019
24
For example, a projection image as shown in FIG. 17 is assumed. In FIG. 17, the horizontal
direction is the x direction, and the vertical direction is the y direction. In the example of FIG. 17,
the coordinates of the upper left vertex of the projection image are (a1, a2), the coordinates of
the upper right vertex are (b1, b2), the coordinates of the lower right vertex are (c1, c2), the
lower left vertex The coordinates are (0, 0). Further, among the coordinates of the five pointed
positions corresponding to the person (person D to the person H) and the five persons reflected
in the display area, the coordinates of the indicated position corresponding to the person D are
(d1, d2) The coordinates of the pointed position corresponding to the person E (e1, e2), the
coordinates of the pointed position corresponding to the person F (f1, f2), the coordinates of the
pointed position corresponding to the person G (g1, g2), the person Let the coordinates of the
designated position corresponding to H be (h1, h2).
[0075]
For example, as shown in FIG. 17, the horizontal position of the cutout line on the right side for
cutting out the unit area including the designated position corresponding to the person G is (b1a1)-(b1-g1) from the left side of the display area. It can be determined as a position separated in
the x direction by +) ((h1−g1) / 2). Further, the horizontal position of the cutout line on the left
side for cutting out the unit area including the designated position corresponding to the person G
is (b1-a1)-(b1-g1)-((g1-) from the left side of the display area. It can be determined as a position
separated in the x direction by f1) / 2).
[0076]
In addition, a method of obtaining a cutout line for cutting out a unit area including a designated
position corresponding to a person H having a designated position adjacent to the left side and a
designated position adjacent to the right side is as follows. The horizontal position of the cutout
line on the right side for cutting out the unit area including the designated position
corresponding to the person H is displayed as (b1-a1)-(b1-h1) + ((h1-g1) / 2). It can be
determined as a position laterally separated from the left side of the area. Further, the horizontal
position of the cutout line on the left side for cutting out the unit area including the designated
position corresponding to the person H is only (b1-a1)-(b1-h1)-((h1-g1) / 2) Can be obtained as a
position laterally separated from the left side of the display area.
[0077]
09-05-2019
25
Further, the method of obtaining the cutout line for cutting out the unit area including the
designated position corresponding to the person D in which the designated position adjacent on
the left side does not exist and the designated position adjacent on the right side is as follows: .
The horizontal position of the cutout line on the right side for cutting out the unit area including
the designated position corresponding to the person D is displayed by (b1-a1)-(b1-d1) + ((e1-d1)
/ 2). It can be determined as a position laterally separated from the left side of the area. Further,
the horizontal position of the cutout line on the left side for cutting out the unit area including
the designated position corresponding to the person D is only (b1-a1)-(b1-d1)-((e1-d1) / 2) Can
be obtained as a position laterally separated from the left side of the display area.
[0078]
Further, the program executed by the conference terminal 2 or the projector 300 described
above is a file of an installable format or an executable format, and is a CD-ROM, a flexible disk
(FD), a CD-R, a DVD (Digital Versatile Disk), It may be configured to be recorded and provided in a
computer readable recording medium such as USB (Universal Serial Bus) or may be configured to
be provided or distributed via a network such as the Internet. In addition, various programs may
be configured to be provided by being incorporated in a ROM or the like in advance.
[0079]
Reference Signs List 1 communication system 2 conference terminal 3 network 4 server 10
panoramic camera 11 display unit 12 microphone array 13 speaker 14 CPU 15 storage device
16 memory 17 LAN I / F unit 18 operation unit 100 network unit 101 control unit 102 display
control unit 111 camera I / F receiver 112 lens characteristic holding unit 113 distortion
correction processing unit 121 first transmission control unit 122 acquisition unit 123
identification unit 124 cutout unit 125 directivity control unit 126 second transmission control
unit 127 output control unit 300 projector 400 stylus pen
[0080]
Patent document 1: JP 2007-274463 A Patent document 5029844
09-05-2019
26
1/--страниц
Пожаловаться на содержимое документа