JP2016502344

Patent Translate
Powered by EPO and Google
Notice
This translation is machine-generated. It cannot be guaranteed that it is intelligible, accurate,
complete, reliable or fit for specific purposes. Critical decisions, such as commercially relevant or
financial decisions, should not be based on machine-translation output.
DESCRIPTION JP2016502344
Generally, techniques for image generation for collaborative sound systems are described. A head
end device comprising a processor can perform these techniques. The processor may be
configured to determine the position of the mobile device participating in the collaborative
surround sound system as a speaker of the multiple speakers of the collaborative surround
sound system. The processor may be further configured to generate an image indicative of the
position of the mobile device participating in the cooperative surround sound system with
respect to the plurality of other speakers of the cooperative surround sound system.
Image generation for collaborative sound systems
[0001]
[0001]This application claims the benefit of US Provisional Application No. 61 / 730,911, filed
Nov. 28, 2012.
[0002]
[0002]The present disclosure relates to multi-channel sound systems, and more particularly to
cooperative multi-channel sound systems.
[0003]
[0003]
A typical multi-channel sound system (which may also be referred to as a "multi-channel
10-05-2019
1
surround sound system") typically includes an audio / video (AV) receiver and two or more
speakers.
An AV receiver typically includes several outputs that interface with the speakers and several
inputs for receiving audio and / or video signals.
Often audio and / or video signals, television sets, digital video disc (DVD) players, high definition
video players, gaming systems, record players, compact disc (CD) players, digital media players,
set top boxes (STBs) , Generated by various home theater components or audio components, such
as laptop computers, tablet computers, etc.
[0004]
[0004]
An AV receiver can process video signals to provide upconversion or other video processing
functionality, but typically an AV receiver will also call the appropriate channel an appropriate
speaker ("loudspeaker") Are used in surround sound systems to perform audio processing to
provide. There are a number of different surround sound formats to reproduce the stage or area
of sound, thereby further providing a more immersive sound experience. In the 5.1 surround
sound system, the AV receiver processes five audio channels, including the center channel, the
left channel, the right channel, the rear right channel, and the rear left channel. The additional
channels that make up the 5.1 ".1" target subwoofer or bus channels. Other surround sound
formats include 7.1 surround sound format (adding additional rear left and rear light channels)
and 22.2 surround sound format (additional forward and rear channels and another subwoofer
or bus channel In addition, additional channels at various heights are added).
[0005]
[0005]
In the context of the 5.1 surround sound format, the AV receiver can process these five channels
and distribute five channels to five loudspeakers and subwoofers. The AV receiver can process
the signal to change the volume level and other characteristics of the signal in order to properly
reproduce the surround sound audio in the particular room in which the surround sound system
operates. That is, the original surround sound audio signal may have been captured and rendered
to fit in a given room, eg, a 15 × 15 foot room. The AV receiver can render this signal to fit in
10-05-2019
2
the room in which the surround sound system operates. The AV receiver can perform this
rendering to create a better sound stage, which can provide a better or even immersive listening
experience.
[0006]
[0006]
Surround sound can provide a more immersive listening experience (and, along with the video, a
visual experience), but the AV receiver and loudspeakers needed to play sound surround sound
are expensive Often. Moreover, in order to properly power the loudspeakers, the AV receiver
often must be physically coupled to the loudspeakers (usually via the loudspeaker wiring). In
order to physically connect the AV receiver to the surround sound system's left speaker and light
rear speaker, considering that surround sound usually requires at least two speakers to be placed
behind the listener In addition, AV receivers often need speaker wiring or other physical
connections to cover the entire room. Fling these wires can be unattractive and prevents
consumers from introducing 5.1, 7.1, and higher order surround sound systems.
[0007]
[0007]
In general, the present disclosure enables collaborative surround sound systems that utilize
mobile devices available as surround sound speakers, or in some instances, front left speakers,
center speakers, and / or front light speakers. Explain the techniques to do this. The head end
device may be configured to perform the techniques described in this disclosure. The head end
device may be configured to interface with one or more mobile devices to form a collaborative
sound system. The head end device can interface with one or more mobile devices to utilize the
speakers of these mobile devices as speakers of a collaborative sound system. Often, head-end
devices may communicate with these mobile devices via a wireless connection, and utilize the
mobile device's speakers for rear left speakers, rear light speakers, or other speakers located
behind the sound system Do.
[0008]
[0008]
In this way, the headend device can form a collaborative sound system using the speakers of the
10-05-2019
3
mobile device that are generally available but not utilized in conventional sound systems, thereby
allowing the user to be dedicated speakers Allows to eliminate or reduce the costs associated
with purchasing. In addition, considering that the mobile device may be wirelessly coupled to the
head end device, the cooperative surround sound system formed in accordance with the
techniques described in the present disclosure may include speaker wiring or other to power the
speakers. The rear sound can be realized without having to look back on the physical connection
of the. Thus, the present technique provides cost savings associated with purchasing a dedicated
speaker and eliminating the expense associated with installing such a speaker, as well as
providing a dedicated physical connection that couples the rear speaker to the headend device.
Both ease of configuration and flexibility by eliminating the need can be encouraged.
[0009]
[0009]In some aspects, a method includes determining a position of a mobile device participating
in a collaborative surround sound system as a speaker of the multiple speakers of the
collaborative surround sound system; Generating an image indicating the position of the mobile
device participating in the cooperative surround sound system with respect to the other
speakers.
[0010]
[0010]In another aspect, the head end device determines the position of the mobile device
participating in the cooperative surround sound system as a speaker of the multiple speakers of
the cooperative surround sound system, and the other of the plurality of cooperative surround
sound systems And a processor configured to generate an image indicative of the position of the
mobile device participating in the cooperative surround sound system with respect to the
speaker.
[0011]
[0011]In another aspect, a headend device comprises means for determining a position of a
mobile device participating in a collaborative surround sound system as a speaker of the multiple
speakers of the collaborative surround sound system; a collaborative surround sound system And
means for generating an image indicative of the position of the mobile device participating in the
cooperative surround sound system with respect to the plurality of other speakers.
[0012]
[0012]
10-05-2019
4
In another aspect, when executed, causes one or more processors to determine the position of
the mobile device participating in the collaborative surround sound system as a speaker of the
multiple speakers of the collaborative surround sound system, the coordination Non-transitory
computer readable storage medium storing instructions for generating an image indicative of the
position of a mobile device participating in a cooperative surround sound system relative to a
plurality of other speakers of a dynamic surround sound system.
[0013]
The details of one or more embodiments of the present technique are set forth in the
accompanying drawings and the description below.
Other features, objects, and advantages of the present technique will be apparent from the
description and drawings, and from the claims.
[0013] [0014]
FIG. 1 is a block diagram illustrating an example collaborative surround sound system formed in
accordance with the techniques described in this disclosure.
[0015]
FIG. 2 is a block diagram illustrating various aspects of the cooperative surround sound system of
FIG. 1 in more detail.
[0016]
FIG. 7 is a flowchart illustrating an example operation of the head end device and the mobile
device in performing the techniques of the collaborative surround sound system described in this
disclosure.
FIG. 7 is a flowchart illustrating an example operation of the head end device and the mobile
device in performing the techniques of the collaborative surround sound system described in this
disclosure. FIG. 7 is a flowchart illustrating an example operation of the head end device and the
mobile device in performing the techniques of the collaborative surround sound system
described in this disclosure.
10-05-2019
5
[0017]
FIG. 7 is a block diagram illustrating further aspects of a cooperative surround sound system
formed in accordance with the techniques described in this disclosure.
[0018]
FIG. 2 is a block diagram illustrating in more detail another aspect of the cooperative surround
sound system of FIG. 1;
[0019]
FIG. 7 shows in more detail an exemplary image as displayed by a mobile device, in accordance
with various aspects of the techniques described in this disclosure. FIG. 7 shows in more detail an
exemplary image as displayed by a mobile device, in accordance with various aspects of the
techniques described in this disclosure. FIG. 7 shows in more detail an exemplary image as
displayed by a mobile device, in accordance with various aspects of the techniques described in
this disclosure.
[0020]
FIG. 7 shows in more detail an exemplary image as displayed by a device coupled to a head end
device, in accordance with various aspects of the techniques described in this disclosure. FIG. 7
shows in more detail an exemplary image as displayed by a device coupled to a head end device,
in accordance with various aspects of the techniques described in this disclosure. FIG. 7 shows in
more detail an exemplary image as displayed by a device coupled to a head end device, in
accordance with various aspects of the techniques described in this disclosure.
[0021]
FIG. 7 is a flowchart illustrating an example operation of the head end device and the mobile
device in performing various aspects of the collaborative surround sound system techniques
described in this disclosure. FIG. 7 is a flowchart illustrating an example operation of the head
end device and the mobile device in performing various aspects of the collaborative surround
sound system techniques described in this disclosure. FIG. 7 is a flowchart illustrating an example
operation of the head end device and the mobile device in performing various aspects of the
collaborative surround sound system techniques described in this disclosure.
[0022]
FIG. 7 is a block diagram illustrating various configurations of a cooperative surround sound
10-05-2019
6
system formed in accordance with the techniques described in this disclosure. FIG. 7 is a block
diagram illustrating various configurations of a cooperative surround sound system formed in
accordance with the techniques described in this disclosure. FIG. 7 is a block diagram illustrating
various configurations of a cooperative surround sound system formed in accordance with the
techniques described in this disclosure.
[0023]
FIG. 7 is a flowchart illustrating an example operation of a head end device in implementing
various power adaptation aspects of the techniques described in this disclosure.
[0024]
FIG. 7 illustrates spherical harmonics based functions of various orders or suborders. FIG. 7
illustrates spherical harmonics based functions of various orders or suborders. FIG. 7 illustrates
spherical harmonics based functions of various orders or suborders.
[0014] [0025] FIG. 1 is a block diagram illustrating an exemplary cooperative surround sound
system 10 formed in accordance with the techniques described in this disclosure. In the example
of FIG. 1, the cooperative surround sound system 10 includes an audio source device 12, a
headend device 14, a front left speaker 16A, a front light speaker 16B, and mobile devices 18A
to 18N ("mobile devices 18"). And. Although shown as including a dedicated front left speaker
16A and a dedicated front light speaker 16B, the present technique is implemented in an
example where mobile device 18 is also used as a front left speaker, center speaker, and front
light speaker It can be done. Thus, the present technique should not be limited to the example
cooperative surround sound system 10 shown in the example of FIG. Moreover, although
described below with respect to cooperative surround sound system 10, the techniques of this
disclosure may be implemented by any form of sound system to provide a cooperative sound
system.
[0015] [0026] Audio source device 12 may represent any type of device capable of generating
source audio data. For example, the audio source device 12 may be a television set (so-called
"smart TV" or "smart TV" running an operating system that features Internet access and / or can
support the execution of applications. Digital set-top box (STB), digital video disc (DVD) player,
high-definition disc player, gaming system, multimedia player, streaming multimedia player,
record player, desktop computer, laptop computer, tablet or tablet Slate computer, cell phone
(including so-called "smart phones"), or capable of generating source audio data, or Can be
provided in a can represent any other type of device or component. In some examples, audio
source device 12 may include a display, such as where audio source device 12 represents a
10-05-2019
7
television, desktop computer, laptop computer, tablet or slate computer, or mobile phone.
[0016] [0027] Head end device 14 represents any device capable of processing (or in other
words rendering) source audio data generated or otherwise provided by audio source device 12.
In some instances, head end device 14 may be an audio source to form a single device, such as,
for example, audio source device 12 being internal to head end device 14 or a portion thereof. It
may be integrated with the device 12. To illustrate, when audio source device 12 represents a
television, desktop computer, laptop computer, slate or tablet computer, gaming system, cell
phone, or high definition disc player, to give a few examples, audio source device 12 May be
integrated with the head end device 14. That is, the headend device 14 may be any of a variety of
devices such as a television, desktop computer, laptop computer, slate or tablet computer,
gaming system, cell phone, high definition disc player, or the like. When the headend device 14 is
not integrated with the audio source device 12, it communicates via either a wired connection or
a wireless connection with the audio source device 12, the front left speaker 16A, the front light
speaker 16B, and / or the mobile device 18. Can represent an audio / video receiver (generally
referred to as an "A / V receiver") that provides several interfaces to do so.
[0017] [0028] Front left speaker 16A and front light speaker 16B ("speaker 16") may represent a
loudspeaker having one or more transducers. Usually, the front left speaker 16A is similar to or
almost the same as the front light speaker 16B. The speaker 16 may comprise a wired interface,
and / or a wireless interface in some instances, to communicate with the head end device 14. The
speakers 16 may be actively powered or may be passively powered, where the head end device
14 drives each of the speakers 16 when passively powered. It can. As mentioned above, the
present technique may be performed without the dedicated speaker 16, which may be replaced
by one or more of the mobile devices 18. In some examples, dedicated speakers 16 may be
incorporated into audio source device 12 or otherwise integrated.
[0018] [0029] Mobile device 18 is typically a mobile phone (including a so-called "smart phone"),
a tablet or slate computer, a netbook, a laptop computer, a digital picture frame, or an application
capable of executing and / or a headend device 14 represents any other type of mobile device
capable of wirelessly interfacing with V.14. Mobile devices 18 may each include speakers 20A20N ("speakers 20"). Each of these speakers 20 may be configured for audio playback, and in
some instances may be configured for audio playback of voice. For ease of illustration, although
the disclosure will be described in the context of a mobile phone, the techniques may be
implemented in any portable device that provides a speaker and can be wired or wireless
communication with the headend device 14 obtain.
10-05-2019
8
[0019] [0030] In a typical multi-channel sound system (which may also be called a "multichannel surround sound system" or "surround sound system"), the A / V receiver, which may
represent a headend device as an example, has its own front left speaker, front center Process
source audio data to match the arrangement of speakers, front light speakers, back left speakers
(sometimes called "surround left speakers"), and back light speakers (sometimes called "surround
light speakers"). A / V receivers often have dedicated wired connections to each of these speakers
to provide better audio quality, power the speakers, and reduce interference. The A / V receiver
may be configured to provide the appropriate channel to the appropriate speaker.
[0020] [0031] There are a number of different surround sound formats to reproduce the stage or
area of sound, thereby further providing a more immersive sound experience. In a 5.1 surround
sound system, the A / V receiver renders five audio channels, including a center channel, a left
channel, a right channel, a rear right channel, and a rear left channel. The additional channels
that make up the 5.1 ".1" target subwoofer or bus channels. Other surround sound formats
include 7.1 surround sound format (adding additional rear left and rear light channels) and 22.2
surround sound format (additional forward and rear channels and another subwoofer or bus
channel In addition, additional channels at various heights are added).
[0021] [0032] In the context of the 5.1 surround sound format, the A / V receiver can render
these five channels for five loudspeakers and the bus channel for the subwoofer. The A / V
receiver can render the signal to change the volume level and other characteristics of the signal
in order to properly reproduce the surround sound audio in the particular room in which the
surround sound system operates. That is, the original surround sound audio signal may be
captured and processed to fit in a given room, eg, a 15 × 15 foot room. The A / V receiver can
process this signal to be compatible with the room in which the surround sound system operates.
The A / V receiver can perform this rendering to create a better sound stage, which can provide a
better or even immersive listening experience.
[0022] [0033] While surround sound can provide a more immersive listening experience (and a
visual experience with the video), the A / V receiver and speakers needed to play sound surround
sound are Often expensive. Moreover, in order to properly power the speakers, the A / V receiver
must be physically coupled to the loudspeakers (usually via the loudspeaker wiring) for the
reasons stated above There are many. Physically connect the A / V receiver to the surround
sound system's Reftoria and light rear speakers, considering that surround sound usually
requires at least two speakers to be placed behind the listener In order to do that, the A / V
receiver often needs speaker wiring or other physical connections to cover the entire room. Fling
these wires can be unattractive and prevents consumers from introducing 5.1, 7.1, and higher
order surround sound systems.
10-05-2019
9
[0023] [0034] According to the techniques described in this disclosure, headend device 14 can
interface with mobile device 18 to form a cooperative surround sound system 10. The head end
device 14 may interface with the mobile device 18 and utilize the speakers 20 of these mobile
devices as surround sound speakers of the cooperative surround sound system 10. Often, the
headend device 14 can communicate with these mobile devices 18 via a wireless connection, as
shown in the example of FIG. 1, a rear left speaker, rear light speaker, or other of the surround
sound system 10 The speaker 20 of the mobile device 18 is used for the speaker placed behind.
[0024] [0035] In this manner, the headend device 14 can form the cooperative surround sound
system 10 using the speakers 20 of the mobile device 18 that are generally available but not
utilized in conventional surround sound systems, thereby Allows the user to eliminate the
expense associated with purchasing a dedicated surround sound speaker. In addition, given that
mobile device 18 may be wirelessly coupled to headend device 14, cooperative surround sound
system 10 formed in accordance with the techniques described in this disclosure is a speaker to
power the speakers Rear surround sound can be achieved without the need for routing or other
physical connections. Thus, the technique saves the cost associated with purchasing a dedicated
surround sound speaker and eliminating the expense associated with installing such a speaker,
and a dedicated physical connection to couple the rear speaker to the headend device. Both ease
of configuration by eliminating the need to provide can be encouraged.
[0025]
[0036]
In operation, the head end device 14 initially includes a mobile device 18 that can include the
corresponding one of the speakers 20 and participate in the cooperative surround sound system
10 (eg, a powered or operable mobile device) 18) can be identified. In some examples, mobile
devices 18 may each execute an application (which may be generally referred to as an "app"),
which coordinates the headend device 18 with the mobile device 18 executing the app. It is
possible to identify as being able to participate in the dynamic surround sound system 10.
[0026]
[0037]
The head end device 14 may then configure the identified mobile device 18 to utilize a
10-05-2019
10
corresponding one of the speakers 20 as one or more speakers of the collaborative surround
sound system 10. In some instances, the head end device 14 may generate source audio data
generated by the audio data source 12 to assist in the configuration of the collaborative surround
sound system 10 (where such source audio data is In the example, the mobile device 18 provides
mobile device data defining the corresponding one aspect of the identified mobile device 18
affecting the audio reproduction of multi-channel audio data ). It can be polled or otherwise
requested. Once in communication with the head end device 14, the mobile device 18
automatically provides this mobile device data when communicating with the head end device 14
and responds to changes to this information without requiring the head end device 14 to request
the information. Mobile device data can be updated regularly. Mobile device 18 may provide
updated mobile device data, for example, when certain aspects of mobile device data change.
[0027]
[0038]
In the example of FIG. 1, mobile device 18 couples wirelessly to headend device 14 via a
corresponding one of sessions 22A-22N ("session 22"), which may also be referred to as
"wireless session 22". The wireless session 22 may be any of the Institute of Electrical and
Electronics Engineers (IEEE) 802.11a standard, IEEE 802.11b standard, IEEE 802.11g standard,
IEEE 802.11n standard, IEEE 802.11ac standard, 802.11ad standard, and any other option. A
wireless session may be provided, such as configured according to, for example, a Personal Area
Network (PAN) standard of the type V. In some instances, the head end device 14 couples to the
wireless network according to one of the standards described above, and the mobile device 18
couples to the same wireless network, and the mobile device 18 often does so. Can register with
the headend device 14 by executing the application and finding the headend device 14 in the
wireless network.
[0028]
[0039]
After establishing the wireless session 22 with the head end device 14, the mobile device 18 can
collect the mobile device data mentioned above, and this mobile device data via the respective
one of the wireless sessions 22 The head end device 14 is provided. The mobile device data may
include any number of characteristics. An exemplary characteristic or aspect defined by the
mobile device data is a corresponding one of the identified mobile devices (using GPS or wireless
network triangulation, if available), the identified mobile device 18 The corresponding one
frequency response of the speakers 20 included in each, the maximum acceptable sound
10-05-2019
11
reproduction level of the speakers 20 included in the corresponding one of the identified mobile
devices 18, the corresponding one of the identified mobile devices 18 Two battery states or
batter power levels, a corresponding synchronization state of the identified mobile device 18 (eg,
whether the mobile device 18 is synchronized with the head end device 14), and the identified
mobile device 18 Corresponding one head Of O emissions state may include one or more.
[0029]
[0040]
Based on this mobile device data, the head end device 14 configures the mobile device 18 to
utilize the speakers 20 of each of these mobile devices 18 as one or more speakers of the
collaborative surround sound system 10 be able to. For example, assuming that mobile device
data defines the location of each of the mobile devices 18, the headend device 14 may not have
one of the identified mobile devices 18 in an optimal location for playing multi-channel audio
source data. May be determined based on the location of one of the mobile devices 18 defined by
the corresponding mobile device data.
[0030]
[0041]
In some examples, one or more of the mobile devices 18 may be responsive to determining that
the head end device 14 is not at a position where one or more of the mobile devices 18 may be
characterized as an optimum position . The cooperative surround sound system 10 can be
configured to control the playback of the audio signal rendered from the audio source in a
manner adapted to multiple sub-optimal locations. That is, the head end device 14 conforms to
the current location of the identified mobile device 18 and provides a more immersive surround
sound experience without the user having to move the mobile device. One or more preprocessing
functions can be configured to render source audio data.
[0031]
[0042]
To further illustrate, the headend device 14 may render the audio signal from the source audio
data so as to effectively change where the audio appears to be generated during playback of the
rendered audio signal. it can. In this sense, the headend device 14 can identify one suitable or
10-05-2019
12
optimal location of the mobile device 18 that is determined to be misaligned, and is referred to as
a virtual speaker of the collaborative surround sound system 10 Establish what you get. The
head end device 14 cross mixes or otherwise distributes the audio signal rendered from the
source audio data, eg, between two or more of the speakers 16 and 20, during playback of the
source audio data. It can create the emergence of such virtual speakers. Further details on how
this audio source data is rendered to create the appearance of virtual speakers are given below
with respect to the example of FIG.
[0032]
[0043]
In this manner, the headend device 14 can identify mobile devices 18 that each include one of
each of the speakers 20 and can participate in the collaborative surround sound system 10. The
head end device 14 may then configure the identified mobile device 18 to utilize each of the
corresponding speakers 20 as one or more virtual speakers of the collaborative surround sound
system. The head end device 14 then causes the audio reproduction of the audio signal to
originate from one or more virtual speakers of the collaborative surround sound system 10 as
the audio signal is reproduced by the speaker 20 of the mobile device 18 As can be seen, audio
signals can be rendered from the audio source data, one or more of their virtual speakers being
different from the position of at least one of the mobile devices 18 (and the corresponding one of
the speakers 20) Often placed in position. Head end device 14 may then transmit the rendered
audio signal to speakers 16 and 20 of cooperative surround sound system 10.
[0033]
[0044]In some instances, the headend device 14 may be configured to effectively optimize
the reproduction of audio signals rendered from multi-channel source audio data by one or more
of the mobile devices 18. One or more users of mobile device 18 may be prompted to relocate
one or more.
[0034]
[0045]
In some examples, head end device 14 may render an audio signal from source audio data based
on mobile device data.
10-05-2019
13
To illustrate, mobile device data can define the power level of the mobile device (which may also
be referred to as "battery status"). Based on this power level, the headend device 14 may use the
audio signal from the source audio data such that some portions of the audio signal have less
loaded audio reproduction (with respect to power consumption to reproduce the audio) It can be
rendered. The head end device 14 may then provide these less loaded audio signals to the mobile
device 18 at reduced power levels. In addition, the head end device 14 is a virtual speaker when
two or more power levels of the mobile device 18 are insufficient to complete playback of the
assigned channel under a known period of source audio data. Determine that two or more of the
mobile devices 18 should cooperate to form a single speaker of the cooperative surround sound
system 10 to reduce power consumption during playback of the audio signal to form a can do.
The above power level adaptations are described in more detail with respect to FIGS. 9A-9C and
FIG.
[0035]
[0046]
The head end device 14 may additionally determine the speaker area in which each of the
speakers of the cooperative surround sound system 10 should be placed. The head end device 14
may then prompt the user in a number of different ways to relocate the corresponding one of the
mobile devices 18 that may be in a sub-optimal position. In one way, the headend device 14
interfaces with the suboptimally located of the mobile devices 18 to be relocated, and those of
the mobile devices 18 are assigned (assigned to more optimal locations) It can indicate the
direction in which the mobile device should be moved, for repositioning, such as in the speaker
area. Alternatively, the headend device 18 can interface with a display such as a television and
present an image identifying the current location of the mobile device and the more optimal
location the mobile device should be moved. The following alternatives for prompting the user to
relocate the sub-optimally located mobile device are in more detail with respect to FIGS. 5, 6A-6C,
7A-7C, and 8A-8C. Described.
[0036]
[0047]
In this manner, the headend device 14 may be configured to determine the position of the mobile
device 18 joining the collaborative surround sound system 10 as a speaker of the multiple
speakers of the collaborative surround sound system 10. Head end device 14 may also be
configured to generate an image that indicates the position of mobile device 18 joining
cooperative surround sound system 10 relative to the plurality of other speakers of cooperative
10-05-2019
14
surround sound system 10.
[0037]
[0048]
However, the headend device 14 can configure the preprocessing function to fit various mobile
devices and situations. For example, the headend device 14 may source audio data based on one
or more characteristics of the speaker 20 of the mobile device 18, eg, the frequency response of
the speaker 20 and / or the maximum acceptable sound reproduction level of the speaker 20. An
audio preprocessing function can be configured to render.
[0038]
[0049]
As yet another example, the headend device 20 receives mobile device data indicating the battery
status or power level of the mobile device 18 utilized as a speaker in the cooperative surround
sound system 10, as described above can do. The headend device 14 may determine that the
power level of one or more of these mobile devices 18 defined by the mobile device data is
insufficient to complete playback of the source audio data. The head end device 14 may then
render the audio signal rendered from the multi-channel source audio data based on the
determination that the power levels of these mobile devices 18 are insufficient to complete
playback of the multi-channel source audio data. The preprocessing function may be configured
to render source audio data to reduce the amount of power required by those of the mobile
devices to play.
[0039]
[0050]
The head end device 14, by way of example, reduces power consumption in these mobile devices
18 by adjusting the volume of audio signals rendered from multi-channel source audio data for
playback by these mobile devices 18. A preprocessing function can be configured. In another
example, the head end device 14 may be configured to transmit audio signals rendered from
multi-channel source audio data to be reproduced by these mobile devices 18 from multi-channel
source audio data to be reproduced by other mobile devices 18. The preprocessing function can
be configured to cross-mix with the rendered audio signal. As yet another example, the head end
10-05-2019
15
device 14 may at least some of the frequencies of the audio signal rendered from the multichannel source audio data to be reproduced by the mobile device 18 lacking sufficient power to
complete the reproduction. The pre-processing function can be configured to reduce the range of
(eg, to remove lower frequencies as an example).
[0040]
[0051]In this manner, the headend device 14 adapts the reproduction of this source audio data
to meet the various needs of the user and to match the various mobile devices 18 and their
corresponding audio capabilities. A preprocessing function can be applied to the source audio
data to allow it to be adapted, or otherwise configured dynamically.
[0041]
[0052]Once the collaborative surround sound system 10 is configured in the various manners
described above, the headend system 14 may then render the rendered audio signal to each of
the one or more speakers of the collaborative surround sound system 10 Can be started, where
one or more of the speakers 20 and / or the speakers 16 of the mobile device 18 can be
coordinated again to form a single speaker of the collaborative surround sound system 10 it can.
[0042]
[0053]
During playback of the source audio data, one or more of the mobile devices 18 may provide
updated mobile device data.
In some examples, the mobile device 18 can stop joining as a speaker of the collaborative
surround sound system 10, to indicate that the corresponding one of the mobile devices 18 is no
longer joining the collaborative surround sound system 10 Provide updated mobile device data.
The mobile device 18 may limit power, set preferences through applications running on the
mobile device 18, receive voice calls, receive email, receive text messages, receive push
notifications, or any number of others. You can stop participating for The headend device 14
may then reorganize the preprocessing function to adapt to changes in the number of mobile
devices 18 participating in the collaborative surround sound system 10. In one example, the
headend device 14 may not prompt the user to move the corresponding mobile device 18 during
playback, and instead, an audio signal that simulates the appearance of virtual speakers in the
10-05-2019
16
manner described above The multi-channel source audio data can be rendered to generate
[0043]
[0054]
In this way, the techniques of this disclosure form an ad hoc network with the central device or
headend system 14 that coordinates the formation of the ad hoc network (which is generally
802.11 or PAN as mentioned above) This effectively enables the mobile device 18 to join the
collaborative surround sound system 10. As described above, the headend device 14 includes
one of the speakers 20 and can join the ad hoc wireless network of the mobile device 18 to play
back the audio signal rendered from the multi-channel source audio data The mobile device 18
can be identified. The head end device 14 then identifies mobile device data defining a
corresponding aspect or characteristic of the identified mobile device 18 that may affect the
audio reproduction of the audio signal rendered from the multi-channel source audio data. Can
be received from each of the selected mobile devices 18. The head end device 14 then controls
the reproduction of the audio signal rendered from the multi-channel source audio data in a
manner compatible with the identified mobile device 18 aspect that affects the audio
reproduction of the multi-channel source audio data Then, an ad hoc wireless network of mobile
devices 18 can be configured based on mobile device data.
[0044]
[0055]
Although described above as directed to a collaborative surround sound system 10 that includes
a mobile device 18 and a dedicated speaker 16, the present technique may be performed on any
combination of mobile device 18 and / or dedicated speakers 16. It can be done. In some
examples, the techniques may be implemented with cooperative surround sound systems that
include only mobile devices. Thus, the technique should not be limited to the example of FIG.
[0045]
[0056]
Moreover, although described throughout as being implemented with respect to multi-channel
source audio data, the present technique is based on object-based audio data and higher order
ambisonic (HOA) audio data The audio data may be defined in the form of hierarchical elements
10-05-2019
17
such as SHC (Spherical Harmonic Coefficient), etc., and may be performed on any type of source
audio data. HOA audio data is described in more detail below with respect to FIGS. 11-13.
[0046]
[0057]
FIG. 2 is a block diagram illustrating a portion of the collaborative surround sound system 10 of
FIG. 1 in more detail. The portion of cooperative surround sound system 10 shown in FIG. 2
includes headend device 14 and mobile device 18A. For ease of illustration, although described
below with respect to a single mobile device, ie, mobile device 18A in the example of FIG. 2, the
present technique includes a plurality of mobile devices, eg, the mobile device shown in the
example of FIG. 18 may be implemented.
[0047]
[0058]
As shown in the example of FIG. 2, the head end device 14 includes a control unit 30. Control
unit 30 (which may also be generally referred to as processor) may represent one or more
central processing units and / or graphical processing units (both not shown in FIG. 2) executing
software instructions, The software instructions are, for example, non-transitory computer
readable storage media (for use in defining software or computer programs) storing instructions
for causing one or more processors to execute the techniques described herein. Also not shown
in FIG. 2), for example, a storage device (eg, a disk drive or an optical drive), or a memory (such
as flash memory, random access memory or RAM), or any other type of volatile memory or Non
volatile It is stored in the directory. Alternatively, control unit 30 may be one or more integrated
circuits, one or more application specific integrated circuits (ASICs), one or more specific
applications for performing the techniques described herein. It may represent dedicated
hardware, such as a special purpose processor (ASSP), one or more field programmable gate
arrays (FPGAs), or any combination of one or more of the above examples of dedicated hardware.
.
[0048]
[0059]
Control unit 30 may execute data retrieval engine 32, power analysis module 34, and audio
10-05-2019
18
rendering engine 36, or may otherwise be configured to implement them. Data retrieval engine
32 may represent a module or unit configured to retrieve, or otherwise receive, mobile device
data 60 from mobile device 18A (and even the remaining mobile devices 18B-18N). The data
retrieval engine 32 may include a position module 38 that determines the position of the mobile
device 18A relative to the headend device 14 when the position is not provided by the mobile
device 18A via mobile device data 62. Data retrieval engine 32 may update mobile device data
60 to include this determined location, thereby generating updated mobile device data 64.
[0049]
[0060]
Power analysis module 34 represents a module or unit configured to process power consumption
data reported by mobile device 18 as part of mobile device data 60. Power consumption data
may include mobile device 18A battery size, audio amplifier rated power, speaker 20A model and
efficiency, and mobile device 18A power profile for different processing (including processing of
wireless audio channels) . Power analysis module 34 may process this power consumption data
to determine refined power data 62, which is returned to data retrieval engine 32. The refined
power data 62 may define the current power level or capacity, the intended power consumption
rate at a given length of time, and the like. Data retrieval engine 32 may then update mobile
device data 60 to include this refined power data 62, thereby generating updated mobile device
data 64. In some instances, power analysis module 34 provides refined power data 62 directly to
audio rendering engine 36, and audio rendering engine 36 updates the refined power data 62 to
the mobile device. In combination with the data 64, the updated mobile device data 64 is further
updated.
[0050]
[0061]
Audio rendering engine 36 represents a module or unit configured to receive updated mobile
device data 64 and process source audio data 37 based on updated mobile device data 64. Audio
rendering engine 36 may process source audio data 37 in any number of ways described in more
detail below. Data retrieval engine 32 and power analysis module 64 are shown as processing
only source audio data 37 with respect to updated mobile device data 64 from a single mobile
device, ie, mobile device 18A in the example of FIG. 2. May retrieve mobile device data 60 from
each of the mobile devices 18 to generate updated mobile device data 64 for each of the mobile
devices 18, and the audio rendering engine 36 may then update the updated mobile device data
Based on each of the 64 examples or combinations of examples (such as when two or more of the
10-05-2019
19
mobile devices 18 are utilized to form a single speaker of the collaborative surround sound
system 10) It is possible to render the Iodeta 37. Audio rendering engine 36 outputs rendered
audio signal 66 for playback by mobile device 18.
[0051]
[0062]
As further shown in FIG. 2, the mobile device 18A includes a control unit 40 and a speaker 20A.
Control unit 40 may be similar to or substantially similar to control unit 30 of head end device
14. Speakers 20A represent one or more speakers by which the mobile device can play source
audio data 37 via playback of the processed audio signal 66.
[0052]
[0063]
Control unit 40 may execute cooperative sound system application 42 and audio playback
module 44 or may otherwise be configured to implement them. A module or unit configured to
establish a wireless session 22A with the headend device 14 and then communicate mobile
device data 60 to the headend device 14 via the wireless session 22A, the collaborative sound
system application 42 Can be represented. The collaborative sound system application 42 also
periodically transmits mobile device data 60 when the collaborative sound system application 42
detects changes in the state of the mobile device 60 that may affect the reproduction of the
rendered audio signal 66. can do. Audio playback module 44 may represent a module or unit
configured to play audio data or signals. Audio playback module 44 may provide rendered audio
signal 66 to speaker 20A for playback.
[0053]
[0064]
The collaborative sound system application 42 may include a data collection engine 46 that
represents a module or unit configured to collect mobile device data 60. Data acquisition engine
46 may include position module 48, power module 50, and speaker module 52. The location
module 48 can determine the location of the mobile device 18A relative to the headend device
14, possibly using a global positioning system (GPS) or through wireless network triangulation.
Often, location module 48 may locate mobile device 18A relative to headend device 14 with
10-05-2019
20
sufficient accuracy to allow headend device 14 to properly perform the techniques described in
this disclosure. It may not be possible.
[0054]
[0065]
In this case, position module 48 may then cooperate with position module 38 executed or
implemented by control unit 30 of head end device 14. Location module 38 may transmit tones
61 or other sounds to location module 48, which interfaces with audio playback module 44 such
that audio playback module 44 causes speaker 20A to play this tone 61. You can take Tone 61
may comprise a tone of a given frequency. Often, the tone 61 is not in the frequency range that is
the cable of what is perceived by the human auditory system. The position module 38 may then
detect the reproduction of this tone 61 by the speaker 20A of the mobile device 18A, and based
on the reproduction of this tone 61, deriving or otherwise determining the position of the mobile
device 18A Can.
[0055]
[0066]
The power module 50 represents a unit or module configured to determine the power
consumption data mentioned above, which power consumption data is again utilized by the
battery size of the mobile device 18A and the audio playback module 44 May include the power
rating of the audio amplifier, the model and power efficiency of the speaker 20A, and the power
profiles of the various processes (including the processing of the wireless audio channel)
performed by the control unit 40 of the mobile device 18A. The power module 50 may determine
this information from system firmware, from an operating system executed by the control unit
40, or from examining various system data. In some examples, the power module 50 can access a
file server or some other data source accessible in a network (such as the Internet), to retrieve
various aspects of this power consumption data Providing the file server with other data
identifying the type, version, manufacturer, or mobile device 18A.
[0056]
[0067]
Speaker module 52 represents a module or unit configured to determine the characteristics of
10-05-2019
21
the speaker. As with the power module 50, the speaker module 52 includes the frequency range
of the speaker 20A, the maximum volume level of the speaker 20A (often expressed in decibels
(dB)), the frequency response of the speaker 20A, etc. Various characteristics can be collected or
otherwise determined. The speaker module 52 may determine this information from system
firmware, from an operating system executed by the control unit 40, or from examining various
system data. In some examples, the speaker module 52 can access a file server or some other
data source accessible in a network (such as the Internet), to retrieve various aspects of this
speaker characterization data Providing the file server with other data identifying the type,
version, manufacturer, or mobile device 18A.
[0057]
[0068]
Initially, as described above, a user or other operator of the mobile device 18A interfaces with
the control unit 40 to execute the collaborative sound system application 42. Control unit 40
executes cooperative sound system application 42 in response to this user input. Assuming that
the collaborative sound system application 42 finds the headend device device 14, the user
interfaces with the collaborative sound system application 42 when running the collaborative
sound system application 42 (often to simplify the illustration The mobile device 18A can be
registered with the headend device 14 (via a touch display presenting a graphical user interface
not shown in the example of FIG. 2). If the headend device 14 can not be found, the collaborative
sound system application 42 can help the user solve any challenges with finding the headend
device 14, and in some cases, for example, the headend device 14. And mobile device 18A
provide troubleshooting information to ensure that both are connected to the same wireless
network or PAN.
[0058]
[0069]
In any event, assuming that the collaborative sound system application 42 finds the headend
device 14 and succeeds in registering the mobile device 18A with the headend device 14, the
collaborative sound system application 42 then generates the data collection engine 46. The
mobile device data 60 can be retrieved and retrieved. In invoking the data acquisition engine 46,
the position module 48 may attempt to determine the position of the mobile device 18A relative
to the headend device 14, and in some cases, the headend device 14 in the manner described
above. The tone 61 is used to coordinate with the location module 38 to enable locating the
mobile device 18 A relative to the head end device 14.
10-05-2019
22
[0059]
[0070]
The tones 61 as described above may also be attempting to coordinate the mobile device 18A
with the position module 38 to determine their position relative to the headend device 14
cooperative surround sound system 10 May be at a given frequency to distinguish it from the
other mobile devices 18B-18N participating in. In other words, the head end device 14 can
associate the mobile device 18A with the tone 61 with the first frequency and the mobile device
18B with the second different frequency, and the third frequency And the mobile device 18C can
be associated, and so on. In this way, rather than finding each of the mobile devices 18
sequentially, the head end device 14 can find multiple mobile devices 18 in parallel
simultaneously.
[0060]
[0071]
The power module 50 and the speaker module 52 can collect power consumption data and
speaker characteristic data in the manner described above. Data collection engine 46 may
aggregate this data to form mobile device data 60. The data acquisition engine 46 includes
mobile device data 60, the location of the mobile device 18A (if possible), the frequency response
of the speaker 20A, the maximum acceptable sound reproduction level of the speaker 20A, the
battery included in the mobile device 18A One of the battery status and power supply to the
mobile device 18A, the synchronization status of the mobile device 18A, and the headphone
status of the mobile device 18A (eg, whether the headphone jack is currently being used to
prevent the use of the speaker 20A) Alternatively, mobile device data 60 can be generated to
define more than one. Data acquisition engine 46 then transmits this mobile device data 60 to
data retrieval engine 32 that is executed by control unit 30 of headend device 14.
[0061]
[0072]
Data retrieval engine 32 may analyze this mobile device data 60 to provide power consumption
data to power analysis module 34. As discussed above, power analysis module 34 may process
this power consumption data to generate refined power data 62. The data retrieval engine 32
10-05-2019
23
may also call the location module 38 to determine the location of the mobile device 18A relative
to the headend device 14 in the manner described above. The data retrieval engine 32 can then
update the mobile device data 60 to include the determined location (if necessary) and the
refined power data 62, and this updated mobile device data 60 Pass to the audio rendering
engine 36.
[0062]
[0073]
Audio rendering engine 36 may then render source audio data 37 based on the updated mobile
device data 64. Audio rendering engine 36 may then configure cooperative surround sound
system 10 to utilize speaker 20A of mobile device 18 as one or more virtual speakers of
cooperative surround sound system 10. The audio rendering engine 36 also causes the audio
reproduction of the rendered audio signal 66 to be at least one of the mobile device 18 such as
the mobile device 18A when the speaker 20A of the mobile device 18A plays the rendered audio
signal 66. Source audio data 37 so as to appear to originate from one or more virtual speakers of
the collaborative surround sound system 10, which often also appear to be located at a different
position from the determined position. An audio signal 66 can be rendered from.
[0063]
[0074]
To illustrate, the audio rendering engine 36 can identify speaker areas where each of the virtual
speakers of the collaborative surround sound system 10 should appear to be generating source
audio data 37. When rendering the source audio data 37, the audio rendering engine 36 then
causes the audio playback of the rendered audio signal 66 to be in the speaker area when the
rendered audio signal 66 is played back by the speaker 20 of the mobile device 18. Audio signal
66 may be rendered from source audio data 37 so as to appear to originate from the virtual
speakers of the collaborative surround sound system 10 at a location in the corresponding
identified one.
[0064]
[0075]
In order to render the source audio data 37 in this manner, the audio rendering engine 36 avoids
10-05-2019
24
prompting the user to move the mobile device 18A by placing it on one of the mobile devices 18,
eg the mobile device 18A. An audio preprocessing function can be configured to render source
audio data 37 based thereon. Considering that moving the mobile device may interfere with
other listeners in the room, avoiding prompting the user to move the device is as after playback
of audio data has begun , May be necessary in some cases. The audio rendering engine 36 then
controls playback of the source audio data in a manner adapted to the location of the mobile
device 18A, using the audio preprocessing function configured when rendering at least a portion
of the source audio data 37 can do.
[0065]
[0076]
In addition, audio rendering engine 36 may render source audio data 37 based on other aspects
of mobile device data 60. For example, audio rendering engine 36 may be based on one or more
speaker characteristics (eg, to match the frequency range of speaker 20A of mobile device 18A
or, as another example, the maximum volume of speaker 20A of mobile device 18A) ), An audio
preprocessing function can be configured for use in rendering source audio data 37. Audio
rendering engine 36 may then render at least a portion of source audio data 37 based on the
configured audio preprocessing function to control playback of rendered audio signal 66 by
speaker 20A of mobile device 18A. .
[0066]
[0077]Audio rendering engine 36 may then send or otherwise transmit rendered audio signal 66
or a portion thereof to mobile device 18.
[0067]
[0078]
3A-3C are flowcharts illustrating exemplary operation of the headend device 14 and the mobile
device 18 in performing the techniques of the cooperative surround sound system described in
this disclosure.
Although described below with respect to a particular one of the mobile devices 18, namely the
mobile device 18A in the example of FIGS. 2 and 3A-3C, the present techniques are similar to
those described herein for the mobile device 18A. May be performed by the mobile devices 18B-
10-05-2019
25
18N.
[0068]
[0079]
Initially, the control unit 40 of the mobile device 18A can execute the collaborative sound system
application 42 (80). The collaborative sound system application 42 may first attempt to find the
presence of the headend device 14 on the wireless network (82). If the collaborative sound
system application 42 can not find the headend device 14 on the network ("No" 84), the mobile
device 18A can continue to try to find the headend device 14 on the network, while Also, in some
cases, troubleshooting information is presented to assist the user in locating the headend device
14 (82). However, if the collaborative sound system application 42 finds the headend device 14
("Yes" 84), then the collaborative sound system application 42 establishes a session 22A and
registers with the headend device 14 via the session 22A. (86), effectively enabling the headend
device 14 to identify the mobile device 18A as a device that can include the speaker 20A and can
participate in the collaborative surround sound system 10.
[0069]
[0080]
After registering with the headend device 14, the collaborative sound system application 42 can
invoke the data acquisition engine 46, which collects mobile device data 60 in the manner
described above (88). . Data collection engine 46 may then transmit mobile device data 60 to
headend device 14 (90). Data retrieval engine 32 of headend device 14 receives mobile device
data 60 (92) and determines whether the mobile device data 60 includes location data defining
the location of mobile device 18 A relative to headend device 14. (94). If the position data is not
sufficient to allow the headend device 14 to accurately position the mobile device 18A (such as
GPS data with only 30 feet accuracy), or if the position data is mobile device data 60 If not ("No"
94), the data retrieval engine 32 can call the position module 38, and the position module 38 can
call the position module 48 of the data collection engine 46 called by the collaborative sound
system application 42. And transmit the tone 61 to the location module 48 of the mobile device
18A (96). Location module 48 of mobile device 18A then passes this tone 61 to audio playback
module 44, which interfaces with speaker 20A to play tone 61 (98).
[0070]
[0081]
10-05-2019
26
On the other hand, after transmitting tone 61, location module 38 of headend device 14 can
interface with a microphone to detect playback of tone 61 by speaker 20A (100). Position
module 38 of headend device 14 may then determine 102 the position of mobile device 18A
based on the detected reproduction of tone 61. After determining the location of mobile device
18A using tone 61, data retrieval module 32 of headend device 18 may update mobile device
data 60 to include the determined location, thereby: Generate updated mobile device data 64
(FIG. 3B, 104).
[0071]
[0082]
When the position data is present in the mobile device data 60 (or the position data is accurate
enough to allow the head end device 14 to position the mobile device 18A relative to the head
end device 14) ) The data retrieval module 32 registers the mobile device 18 with the headend
device 14 if the data retrieval module 32 determines or after generating the updated mobile
device data 64 to include the determined position. It may be determined whether it has finished
retrieving mobile device data 60 from each of (106). If the data retrieval module 32 of the
headend device 14 has not finished retrieving the mobile device data 60 from each of the mobile
devices 18 ("No" 106), the data retrieval module 32 proceeds in the manner described above (
92-106), continuing to retrieve mobile device data 60 and generate updated mobile device data
64. However, if the data retrieval module 32 determines that it has finished collecting the mobile
device data 60 and generating the updated mobile device data 64 ("Yes" 106), the data retrieval
module 32 updates The mobile device data 64 is passed to the audio rendering engine 36.
[0072]
[0083]
Audio rendering engine 36 may retrieve source audio data 37 in response to receiving this
updated mobile device data 64 (108). When rendering the source audio data 37, the audio
rendering engine 36 may first determine a speaker area, which represents the area where the
speakers should be arranged to fit the playback of the multi-channel source audio data 37. (110).
For example, 5.1 channel source audio data includes a front left channel, a center channel, a front
light channel, a surround left channel, a surround light channel, and a subwoofer channel. The
subwoofer channel is not directional or worth noting considering that low frequencies usually
provide a sufficient effect regardless of the position of the subwoofer relative to the head end
10-05-2019
27
device. However, the other five channels, however, may correspond to specific locations to
provide the best sound stage for immersive audio reproduction. The audio rendering engine 36
may interface with the location module 38 to derive room boundaries in some instances, such
that the location module 38 locates walls, people, furniture, etc. , Speaker 16 and / or speaker 20
may emit tone or sound. Based on the room or object position information, the audio rendering
engine 36 can determine the speaker area for each of the front left speaker, the center speaker,
the front light speaker, the surround left speaker, and the surround light speaker.
[0073]
[0084]
Based on these speaker areas, audio rendering engine 36 may determine the position of virtual
speakers of collaborative surround sound system 10 (112). That is, the audio rendering engine
36 can place a virtual speaker in each of the speaker areas that are often at optimal or nearoptimal positions for room or object position information. Audio rendering engine 36 may then
assign mobile device 18 to each virtual speaker based on mobile device data 18 (114).
[0074]
[0085]
For example, the audio rendering engine 36 may first consider the location of each of the mobile
devices 18 defined in the updated mobile device data 60, and those devices are closest to the
determined location of the mobile device 18 Assign to a virtual speaker having a virtual position.
The audio rendering engine 36 may determine whether to assign more than one of the mobile
devices 18 to a virtual speaker based on how close the currently assigned mobile device 18 is to
the position of the virtual speaker. Moreover, as described above, the audio rendering engine 36
is unable to reproduce the entire source audio data 37 by the refined power data 62 associated
with one of the two or more mobile devices 18 When sufficient, it may be decided to assign two
or more of the mobile devices 18 to the same virtual speaker. Also as described above, audio
rendering engine 36 may also assign these mobile devices 18 based on other aspects of mobile
device data 60 including speaker characteristics.
[0075]
[0086]
10-05-2019
28
The audio rendering engine 36 may then render audio signals from the source audio data 37 in
the manner described above for each of the speakers 16 and 20, based on the virtual speaker
location and / or the mobile device data 60. , Effectively render the audio signal (116). In other
words, audio rendering engine 36 may then instantiate or otherwise define preprocessing
functions to render source audio data 37, as described in more detail above. In this manner,
audio rendering engine 36 may render or otherwise process source audio data 37 based on the
virtual speaker location and mobile device data 60. As mentioned above, when the audio
rendering engine 36 processes this audio data, it considers the mobile device data 60 from each
of the mobile devices 18 as a whole or as a whole, and is still rendered from the audio source
data 60 Separate audio signals may be sent to each of the mobile devices 18. Thus, the audio
rendering engine 36 sends the rendered audio signal 66 to the mobile device 18 (FIG. 3C, 120).
[0076]
[0087]
In response to receiving this rendered audio signal 66, the collaborative sound system
application 42 interfaces with the audio playback module 44, which is then rendered interfaced
with the speaker 20A. The audio signal 66 is played back (122). As mentioned above, the
collaborative sound system application 42 can periodically call the data collection engine 46 to
determine if any of the mobile device data 60 has changed or has been updated. (124). If the
mobile device data 60 has not changed ( No 124), the mobile device 18A continues to play
the rendered audio signal 66 (122). However, if mobile device data 60 has changed or has been
updated ("Yes" 124), data collection engine 46 transmits this modified mobile device data 60 to
data retrieval engine 32 of headend device 14. It can be done (126).
[0077]
[0088]
The data retrieval engine 32 can pass this modified mobile device data to the audio rendering
engine 36, which based on the modified mobile device data 60, mobile device via the
construction of a virtual speaker A preprocessing function can be modified to render the audio
signal to which 18A is assigned. As will be described in more detail below, the mobile device data
60 that is generally updated or changed is, by way of example, a change in power consumption
or another voice call such as to interrupt audio playback. It changes because the mobile device
18A has been used for the task.
10-05-2019
29
[0078]
[0089]
In some examples, data retrieval engine 32 may determine that mobile device data 60 has
changed in the sense that location module 38 of data retrieval module 32 may detect a change in
the position of mobile device 18. In other words, the data retrieval module 32 can periodically
call the location module 38 to determine the current location of the mobile device 18 (or
alternatively, the location module 38 continuously locates the location of the mobile device 18
Can be monitored). The location module 38 may then determine if one or more of the mobile
devices 18 have been moved, thereby causing the audio rendering engine 36 to generate
ongoing changes in the location of the mobile devices 18 (eg, the user It enables the
preprocessing function to be dynamically modified to match the mobile device to get the text
message and then fit the mobile device to different locations. Thus, the present technique may
possibly ensure that the virtual speakers stay at least near the optimal position during the entire
playback, even though the mobile device 18 may be moved or repositioned during playback. It
can be applied in a dynamic scene.
[0079]
[0090]
FIG. 4 is a block diagram illustrating another cooperative surround sound system 140 formed in
accordance with the techniques described in this disclosure. In the example of FIG. 4, audio
source device 142, headend device 144, front left speaker 146A, front light speaker 146B, and
mobile devices 148A-148C are described above with respect to FIGS. 1, 2, 3A-3C, respectively.
Audio source device 12, head end device 14, front left speaker 16A, front light speaker 16B, and
mobile devices 18A-18N may be substantially similar.
[0080]
[0091]
As shown in the example of FIG. 4, the headend device 144 divides the room in which the
collaborative surround sound system 140 operates into five separate speaker areas 152A-152E
("area 152"). After determining these areas 152, head end device 144 may determine the
position of virtual speakers 154A-154E ("virtual speakers 154") relative to each of areas 152.
10-05-2019
30
[0081]
[0092]
For each of the regions 152A and 152B, the head end device 144 determines that the positions
of the virtual speakers 154A and 154B are close to or coincident with the positions of the front
left speaker 146A and the front light speaker 146B, respectively. For region 152C, headend
device 144 determines that the position of virtual speaker 154C does not overlap with any of
mobile devices 148A-148C ("mobile device 148"). As a result, head end device 144 searches
region 152C to identify any mobile devices 148 located within or partially within region 152C. In
performing this search, headend device 144 determines that mobile devices 148A and 148B are
located within region 152C, or at least partially within. Head end device 144 then assigns these
mobile devices 148A and 148B to virtual speaker 154C. The head end device 144 then performs
a first pre-processing function to render the surround left channel from the source audio data for
playback by the mobile device 148A so that it appears as if the sound originates from the virtual
speaker 154C. Define Head end device 144 also causes a second front of the surround light
channel to render a second instance of the surround light channel from the source audio data for
playback by mobile device 148B as if sound were originating from virtual speaker 154C. Define a
processing function.
[0082]
[0093]
The head end device 144 may then consider the virtual speaker 154D, saying that the position of
the mobile device 148C overlaps with the position of the virtual speaker 154D (often within a
defined or set threshold). In terms of points, it is determined that the mobile device 148C is
located at a position near the optimum in the region 152D. Head end device 144 may define a
pre-processing function to render the surround light channel based on other aspects of the
mobile device data associated with mobile device 148 C, but this surround light channel is
generated It may not be necessary to define a preprocessing function to correct where it appears
to be.
[0083]
[0094]
Head end device 144 may then determine that a center speaker that can support virtual speaker
154E is not within center speaker area 152E. As a result, the head end device 144 renders the
10-05-2019
31
center channel from the source audio data so that the front left speaker 146A and the front light
speaker 146B play both their respective left front channels and the front light channel and the
center channel. A pre-processing function can be defined that cross mixes both the front left
channel and the front light channel with the center channel. This pre-processing function can
modify the center channel so that the sound appears as if it were being played from the position
of the virtual speaker 154E.
[0084]
[0095]
When defining preprocessing functions that process source audio data such that the source
audio data appears to originate from virtual speakers such as virtual speaker 154C and virtual
speaker 154E, one or more of the speakers 150 may When not located at the intended position
of the virtual speaker, the head end device 144 may perform aspects of dynamic amplitude
panning based on constraint vectors of the techniques described in this disclosure. Instead of
performing vector-based amplitude panning (VBAP) based on only a pair of speakers (two
speakers for two dimensions and three speakers for three dimensions), the headend device 144 A
constraint vector based dynamic amplitude panning technique can be performed for more than
two speakers. Constraint vector based dynamic amplitude panning techniques can be based on
realistic constraints, thus providing a high degree of freedom compared to VBAP.
[0085]
[0096]
To illustrate, consider the following example where three loudspeakers may be located at the
rear left corner (and thus in the surround left speaker area 152C). In this example, given [p 1 p
2] <T>, the vector represents the power and position of the virtual source, [l 11 l 12] <T>, [l 21 l
22] <T> Three vectors may be defined, which may be denoted by [l 31 l 32] <T>. The head end
device 144 can then solve the following equation:
[0086]
ここで、
[0087]
Are unknowns that the head end device 144 may need to calculate.
10-05-2019
32
[0088]
[0097]
[0089]
Solving is a typical multi-unknown problem, and the usual solution involves the headend device
144 finding a least norm solution.
Assuming that the head end device 144 solves this equation using the L2 norm, the head end
device 144 solves the following equation:
[0090]
[0098]
The headend device 144 can constrain g 1, g 2 and g 3 in some way by manipulating the vectors
based on the constraints.
Head end device 144 may then add scalar power coefficients a 1, a 2, and a 3 as follows.
[0091]
、かつ
[0092]
[0099]When using an L2-norm solution, which is a solution that provides adequate gain for each
of the three speakers located in the surround left region 152C, the head end device 144
produces virtually placed loudspeakers At the same time, the power sum of the gains allows the
headend device 144 to properly distribute the power consumption for all three available
loudspeakers under the constraints on the inherent power consumption limits, Note that it is
minimal.
10-05-2019
33
[0093]
[0100]
To illustrate, if the battery power of the second device is being depleted, the headend device 144
can reduce a 2 relative to the other powers a 1 and a 3.
As a more specific example, headend device 144 has three loudspeaker vectors [1 0] <T>,
[0094]
, [1 0] <T> and the headend device 144
[0095]
It is assumed that it is constrained to have
If there is no constraint meaning a 1 = a 2 = a 3 = 1
[0096]
である。
However, for some reasons, such as a battery or a specific maximum volume per loudspeaker, the
headend device 144 may need to reduce the volume of the second loudspeaker, and the second
vector
[0097]
Only by
[0098]
10-05-2019
34
となる。
In this example, the head end device 144 may lower the gain of the second loudspeaker, but still
the virtual image remains the same, or nearly remains at the same position.
[0099]
[0101]These techniques described above can be generalized as follows.
[0100]
1.
If the headend device 144 determines that one or more of the speakers have frequency
dependent constraints, the headend device may, via any type of filter bank analysis and
synthesis, including a short time Fourier transform
[0101]
The above equation can be defined to depend on k, where k is the frequency index.
[0102]
2.
The head end device 144 can extend this to the case of any N ≧ 2 loudspeakers by allocating a
vector based on the detected position.
[0103]
3.
10-05-2019
35
The head end device 144 can arbitrarily group any combination with appropriate power gain
constraints, which may or may not be overlapping. In some examples, the head end device 144
can use all loudspeakers simultaneously to generate sounds based on five or more different
locations. In some examples, the head end device 144 can group loudspeakers in each designated
area, eg, five speaker areas 152 shown in FIG. If there is only one in one area, head end device
144 can expand the group for that area to the next area.
[0104]
4. If some devices are moving or are only registered with the cooperative surround sound
system 140, the headend device 144 updates (changes or adds) the corresponding basic vectors
to each speaker The gain for can be calculated and this gain is likely to be adjusted.
[0105]
5. Although described above with respect to the L2 norm, the headend device 144 can utilize
different norms other than the L2 norm to solve for this minimum norm. For example, when
using the L0 norm, the head end device 144 can calculate a low gain solution, which means that
the small gain loudspeaker for L2 norm will be a zero gain loudspeaker Do.
[0106]
6. The power constrained minimum norm solution presented above is a specific way to
implement the constraint optimization problem. However, any kind of constrained convex
optimization method
[0107]
It can be combined with the problem of
[0108]
[0102]
10-05-2019
36
In this manner, the headend device 144 can locate the defined position of the virtual speaker
154C of the collaborative surround sound system 140 relative to the mobile device 150A joining
the collaborative surround sound system 140.
The head end device 144 can then determine constraints that affect the playback of the multichannel audio data by the mobile device, such as the expected power duration. The head end
device 144 then uses the determined constraints to render the audio signal 66 in a manner that
reduces the effect of the determined constraints on the reproduction of the rendered audio signal
66 by the mobile device 150A. Dynamic amplitude panning based on the constraint vectors
described above for data 37 can be performed.
[0109]
[0103]
In addition, the head end device 144 determines an expected power period that indicates the
expected period when the mobile device will have sufficient power to play the source audio data
37 when determining constraints. be able to. The head end device 144 may then determine a
source audio period that indicates the playback period of the source audio data 37. When the
source audio period exceeds the expected power period, the headend device 144 can determine
the expected power period as a constraint.
[0110]
[0104]Moreover, in some examples, when performing constraint amplitude based dynamic
amplitude panning, the headend device 144 has an expected power duration for reproducing the
rendered audio signal 66 that is shorter than the source audio duration. As such, to render audio
signal 66, dynamic amplitude panning based on constraint vectors can be performed on source
audio data 37 using the determined expected power duration as a constraint.
[0111]
[0105]
In some instances, when determining constraints, head end device 144 may determine frequency
dependent constraints.
10-05-2019
37
When performing dynamic amplitude panning based on a constraint vector, the head end device
144 determines the reproduction period of the source audio data 37 as an example of the
expected power period for reproducing the audio signal 66 rendered by the mobile device 150A.
In order to render the audio signal 66 to be shorter than the indicated source audio period,
dynamic amplitude panning based on constraint vectors can be performed on the source audio
data 37 using the determined frequency constraints .
[0112]
[0106]
In some examples, when performing constraint amplitude based dynamic amplitude panning,
headend device 144 may consider multiple mobile devices supporting one of multiple virtual
speakers. As mentioned above, in some examples, headend device 144 may perform this aspect
of the present technique for three mobile devices. When performing dynamic amplitude panning
based on a constraint vector for source audio data 37 using the expected power duration as a
constraint and assuming that three mobile devices support a single virtual speaker, headend
device 144 First, for each of the first mobile device, the second mobile device, and the third
mobile device, the volume gains g 1, g 2 and g 3 can be calculated according to the following
equations.
[0113]
[0107]
As mentioned above, a 1, a 2 and a 3 are the scalar power factor for the first mobile device, the
scalar power factor for the second mobile device and the scalar power factor for the third mobile
device Show. l 11 and l 12 indicate vectors specifying the position of the first mobile device
relative to the head end device 144. l 21 and l 22 indicate vectors specifying the position of the
second mobile device with respect to the head end device 144. l 31 and l 32 indicate vectors
specifying the position of the third mobile device relative to the head end device 144. p 1, p 2 are
vectors that specify a defined position with respect to the head end device 144 of one of a
plurality of virtual speakers supported by the first mobile device, the second mobile device, and
the third mobile device Indicates
[0114]
10-05-2019
38
[0108]
FIG. 5 is a block diagram illustrating a portion of the collaborative surround sound system 10 of
FIG. 1 in more detail. The portion of cooperative surround sound system 10 shown in FIG. 2
includes headend device 14 and mobile device 18A. For ease of illustration, although described
below with respect to a single mobile device, ie, mobile device 18A in the example of FIG. 5, the
present technique includes multiple mobile devices, eg, the mobile device shown in the example
of FIG. 18 may be implemented.
[0115]
[0109]
As shown in the example of FIG. 5, the head end device 14 includes the same components, units
and modules as described and shown above with respect to the example of FIG. 2 that also
includes an additional image generation module 160. The image generation module 160
generates one or more images 170 for display via the display device 164 of the mobile device
18A, and one or more for display via the display device 166 of the source audio device 12. The
module or unit is configured to generate an image 172 of Image 170 may represent any one or
more images that may define the direction or position in which mobile device 18A should be
moved or positioned. Similarly, images 172 may represent one or more images indicating the
current position of mobile device 18A and the desired or intended position of mobile device 18A.
Image 172 may also define the direction in which mobile device 18A should be moved.
[0116]
[0110]
Similarly, mobile device 18A includes the same components, units, and modules as described and
shown above with respect to the example of FIG. 2 that also includes display interface module
168. Display interface module 168 may represent a unit or module of collaborative sound system
application 42 configured to interface with display device 164. Display interface module 168
may interface with display device 164 to transmit image 170 or otherwise cause display device
164 to display image 170.
[0117]
[0111]
10-05-2019
39
Initially, as described above, the user of the mobile device 18A or other operator interfaces with
the control unit 40 to execute the collaborative sound system application 42. Control unit 40
executes cooperative sound system application 42 in response to this user input. Assuming that
the collaborative sound system application 42 finds the headend device device 14, the user
interfaces with the collaborative sound system application 42 when running the collaborative
sound system application 42 (often to simplify the illustration The mobile device 18A can be
registered with the headend device 14 (via a touch display presenting a graphical user interface
not shown in the example of FIG. 2). If the headend device 14 can not be found, the collaborative
sound system application 42 can help the user solve any challenges with finding the headend
device 14, and in some cases, for example, the headend device 14. And mobile device 18A
provide troubleshooting information to ensure that both are connected to the same wireless
network or PAN.
[0118]
[0112]
In any event, assuming that the collaborative sound system application 42 finds the headend
device 14 and succeeds in registering the mobile device 18A with the headend device 14, the
collaborative sound system application 42 then generates the data collection engine 46. The
mobile device data 60 can be retrieved and retrieved. In invoking the data acquisition engine 46,
the position module 48 may attempt to determine the position of the mobile device 18A relative
to the headend device 14, and in some cases, the headend device 14 in the manner described
above. The tone 61 is used to coordinate with the location module 38 to enable locating the
mobile device 18 A relative to the head end device 14.
[0119]
[0113]
The tones 61 as described above may also be attempting to coordinate the mobile device 18A
with the position module 38 to determine their position relative to the headend device 14
cooperative surround sound system 10 May be at a given frequency to distinguish it from the
other mobile devices 18B-18N participating in. In other words, the head end device 14 can
associate the mobile device 18A with the tone 61 with the first frequency and the mobile device
18B with the second different frequency, and the third different frequency Can be associated
with the mobile device 18C, and so on. In this manner, the headend device 14 can simultaneously
10-05-2019
40
find multiple mobile devices 18 in parallel, rather than finding each of the mobile devices 18
sequentially.
[0120]
[0114]
The power module 50 and the speaker module 52 can collect power consumption data and
speaker characteristic data in the manner described above. Data collection engine 46 may
aggregate this data to form mobile device data 60. The data acquisition engine 46 includes the
location of the mobile device 18A (if possible), the frequency response of the speaker 20A, the
maximum acceptable sound reproduction level of the speaker 20A, the battery status of the
battery contained in the mobile device 18A and the mobile device 18A. Define one or more of:
power on, synchronization status of the mobile device 18A, and headphone status of the mobile
device 18A (eg, whether the headphone jack is currently being used to prevent the use of the
speaker 20A), Mobile device data 60 can be generated. Data acquisition engine 46 then transmits
this mobile device data 60 to data retrieval engine 32 that is executed by control unit 30 of
headend device 14.
[0121]
[0115]
Data retrieval engine 32 may analyze this mobile device data 60 to provide power consumption
data to power analysis module 34. As discussed above, power analysis module 34 may process
this power consumption data to generate refined power data 62. The data retrieval engine 32
may also call the location module 38 to determine the location of the mobile device 18A relative
to the headend device 14 in the manner described above. The data retrieval engine 32 can then
update the mobile device data 60 to include the determined location (if necessary) and the
refined power data 62, and this updated mobile device data 60 Pass to the audio rendering
engine 36.
[0122]
[0116]
Audio rendering engine 36 may then process source audio data 37 based on the updated mobile
device data 64. The audio rendering engine 36 may then configure the collaborative surround
10-05-2019
41
sound system 10 to utilize the speaker 20A of the mobile device 18A as one or more virtual
speakers of the collaborative surround sound system 10. The audio rendering engine 36 also
causes the audio reproduction of the rendered audio signal 66 to be at a position different from
the determined position of the mobile device 18A when the speaker 20A of the mobile device
18A reproduces the rendered audio signal 66. The audio signal 66 may be rendered from the
source audio data 37 so as to appear to originate from one or more virtual speakers of the
collaborative surround sound system 10, which often appear to be deployed. .
[0123]
[0117]
To illustrate, the audio rendering engine 36 receives one or more of the collaborative surround
sound system 10 when given the mobile device data 60 from one or more of the mobile devices
18 supporting the corresponding one or more of the virtual speakers. A speaker area can be
assigned to each one of the virtual speakers of. When rendering the source audio data 37, the
audio rendering engine 36 then performs audio playback of the rendered audio signal 66 as the
rendered audio signal 66 is played back by the speaker 20 of the mobile device 18. The source
so that it appears to originate from the virtual speakers of the collaborative surround sound
system 10 which are also often located in the corresponding identified one of the different
speaker areas than the at least one position of the An audio signal 66 can be rendered from
audio data 37.
[0124]
[0118]
In order to render the source audio data 37 in this manner, the audio rendering engine 36 avoids
prompting the user to move the mobile device 18A by placing it on one of the mobile devices 18,
eg the mobile device 18A. An audio preprocessing function can be configured to render source
audio data 37 based thereon. In some instances, such as when initially placing the mobile device
18 in the room before playback, after the playback of the audio signal 66 has begun, it is
necessary to avoid user prompts to move the device Although possible, the head end device 14
may prompt the user to move the mobile device 18 in some instances. The head end device 14
analyzes the speaker area and determines one or more of the mobile devices 18 by determining
that one or more areas do not have any mobile devices or other speakers present in the area. It
can be determined that it needs to be moved.
10-05-2019
42
[0125]
[0119]
The head end device 14 then determines whether any speaker area has more than one speaker,
and, based on the updated mobile device data 64, whichever of these two or more speakers is
this speaker area It can be specified whether it should be relocated to an empty speaker area that
does not have the mobile device 18 located inside. The head end device 14 may consider the
refined power data 62 when attempting to relocate one or more of the two or more speakers
from one speaker area to another, and two Of the above speakers, it is decided to rearrange a
speaker having at least sufficient power to reproduce the whole of the rendered audio signal 66,
as indicated by the refined power data 62. If there are no speakers meeting this power criteria,
the head end device 14 may have two or more speakers from the overloaded speaker area (which
may point to a speaker area having two or more speakers located in that area). Two or more
speakers can be determined to the empty speaker area (which may point to a speaker area where
no mobile device or other speakers are present).
[0126]
[0120]
Having determined which of the mobile devices 18 are to be relocated to the empty speaker area
and the position at which these mobile devices 18 are to be deployed, the control unit 30 may
call the image generation module 160 it can. The location module 38 may provide the imaging
module 160 with the intended or desired location and the current location of the mobile device
18 to be relocated. Image generation module 160 may then generate images 170 and / or 172,
and transmit these images 170 and / or 172 to mobile device 18A and source audio device 12,
respectively. Mobile device 18A may then present image 170 via display device 164, while
source audio device 12 may present image 172 via display device 164. Image generation module
160 may continue to receive updates to the current position of mobile device 18 from position
module 38 and continue to generate images 170 and 172 representing the updated current
position. In this sense, the image generation module 160 can dynamically generate images 170
and / or 172 that reflect the current movement of the mobile device 18 relative to the head end
unit 14 and the intended position. Once in the intended position, the image generation module
160 can generate images 170 and / or 172 that indicate that the mobile device 18 has been
placed in the intended or desired position, , To facilitate configuration of the cooperative
surround sound system 10. Images 170 and 172 are described in more detail below with respect
to FIGS. 6A-6C and 7A-7C.
10-05-2019
43
[0127]
[0121]
Additionally, audio rendering engine 36 may render audio signal 66 from source audio data 37
based on other aspects of mobile device data 60. For example, audio rendering engine 36 may be
based on one or more speaker characteristics (eg, to match the frequency range of speaker 20A
of mobile device 18A or, as another example, the maximum volume of speaker 20A of mobile
device 18A) ), An audio pre-processing function can be configured to render source audio data
37. The audio rendering engine 36 may then apply the configured audio preprocessing function
to at least a portion of the source audio data 37 to control playback of the rendered audio signal
66 by the speaker 20A of the mobile device 18A. it can.
[0128]
[0122]
Audio rendering engine 36 may then send or otherwise transmit rendered audio signal 66 or a
portion thereof to mobile device 18A. Audio rendering engine 36 may assign one or more of
mobile devices 18 to each channel of multi-channel source audio data 37 via a virtual speaker
configuration. That is, each of the mobile devices 18 is assigned to different virtual speakers of
the collaborative surround sound system 10. Each virtual speaker is then assigned to a speaker
area, which may support one or more channels of multi-channel source audio data 37. Thus,
when transmitting the rendered audio signal 66, the audio rendering engine 36 configures the
mapped channels of the rendered audio signal 66 as the corresponding one or more virtual
speakers of the collaborative surround sound system 10 Can be sent to the corresponding one or
more of the mobile devices 18 that have been sent.
[0129]
[0123]
Throughout the discussion of the techniques described below with respect to FIGS. 6A-6C and
7A-7C, references to channels may be as follows. That is, the left channel may be denoted as "L",
the light channel may be denoted as "R", the center channel may be denoted as "C", and the rear
left channel is "surround left channel" And may be designated as "SL", the rear light channel may
be referred to as "surround light channel", and may be designated as "SR". Again, the subwoofer
channel is not shown in FIG. 1 as the subwoofer's position is not as important as the other five
channel positions in providing a good surround sound experience.
10-05-2019
44
[0130]
[0124]
6A-6C are diagrams showing the example images 170A-170C of FIG. 5 in more detail as
displayed by the mobile device 18A, in accordance with various aspects of the techniques
described in this disclosure. FIG. 6A is a diagram showing a first image 172A including an arrow
173A. Arrow 173A indicates the direction in which mobile device 18A should be moved to place
mobile device 18A in the intended or optimal position. The length of arrow 173A may
schematically indicate how far away the current position of mobile device 18A is from the
intended position.
[0131]
[0125]
FIG. 6B is a diagram showing a second image 170B including a second arrow 173B. Arrow 173B
may indicate, as arrow 173A, the direction in which mobile device 18A should be moved to
position mobile device 18A in the intended or optimal position. Arrow 173B differs from arrow
173A in that it is shorter in length, which is closer to the intended position of mobile device 18A
with respect to the position of mobile device 18A when image 170A is presented. Indicates that it
has moved. In this example, image generation module 160 may generate image 170B in response
to location module 38 providing the updated current location of mobile device 18A.
[0132]
[0126]
FIG. 6C is a diagram showing a third image 170C, which may be referred to as image 170 (which
is shown in the example of FIG. 5). Image 170C indicates that mobile device 18A has been placed
at the intended location of the surround left virtual speaker. Image 170C includes an indication
174 ("SL") that mobile device 18A has been placed at the intended location of the surround left
virtual speaker. Image 170C also re-arranges the device as a surround sound back left speaker so
that the user further understands that mobile device 18 is properly positioned at the location
intended to support the virtual surround sound speaker. Include a text area 176 indicating The
user confirms (button 178A) or cancels (button 178B) that the image 170C additionally registers
with the mobile device 18A as an add-on to support the surround sound left virtual speakers of
10-05-2019
45
the collaborative surround sound system 10. And includes two virtual buttons 178A and 178B.
[0133]
[0127]
7A-7C are diagrams showing the example images 172A-172C of FIG. 5 in more detail as
displayed by the source audio device 12, in accordance with various aspects of the techniques
described in this disclosure. FIG. 7A includes a first image 170A including speaker areas 192A192E, speakers 194A-194E (which may represent mobile device 18), an indication 196 of the
intended surround sound virtual speaker left, and an arrow 198A. FIG. The speaker areas 192A192E ("speaker area 192") may each represent different speaker areas of the 5.1 surround sound
format. Although illustrated as including five speaker areas, the present technique may be
applied to any of the speaker areas, including seven speaker areas to match the 7.1 surround
sound format and the emerging three-dimensional surround sound format. It may be
implemented in terms of configuration.
[0134]
[0128]
Speakers 194A-194E ("speaker 194") may represent the current position of speaker 194, where
speaker 194 may represent speaker 16 and mobile device 18 shown in the example of FIG. When
properly positioned, the speaker 194 may represent the intended position of the virtual speaker.
If it detects that one or more of the speakers 194 is not properly positioned to support one of the
virtual speakers, the headend device 14 can generate an image 172A, and the arrow 198A is one
of the speakers 194 Indicates that one or more should be moved. In the example of FIG. 7A, the
mobile device 18A represents a surround sound left (SL) speaker 194C that is located outside the
surround light (SR) speaker area 192D. Thus, the head end device 14 generates an image 172A
with an arrow 198A indicating that the SL speaker 194C is to be moved to the intended SL
position 196. The intended SL position 196 represents the intended position of the SL speaker
194C, where the arrow 198A points from the current position of the SL speaker 194C towards
the intended SL position 196. The headend device 14 may also generate the image 170A
described above for display on the mobile device 18A to further facilitate relocation of the mobile
device 18A.
[0135]
[0129]
10-05-2019
46
FIG. 7B is a diagram showing a second image 172B similar to the image 172A except that the
image 172B includes a new arrow 198B and the current position of the SL speaker 194C has
moved to the left. Arrow 198B, as arrow 198A, may indicate the direction in which mobile device
18A should be moved to place mobile device 18A in the intended position. Arrow 198B differs
from arrow 198A in that it is shorter in length, which is closer to the intended position of mobile
device 18A with respect to the position of mobile device 18A when image 172A is presented.
Indicates that it has moved. In this example, image generation module 160 may generate image
172B in response to location module 38 providing the updated current location of mobile device
18A.
[0136]
[0130]
FIG. 7C is a diagram showing a third image 172C, which may be referred to as image 172 (which
is shown in the example of FIG. 5). Image 172C indicates that mobile device 18A has been placed
at the intended location of the surround left virtual speaker. Image 170C removes the indication
196 of the intended position and indicates that the SL speaker 194C is properly positioned
(removing the dotted line of the SL indication 196 and replacing it with a solid SL speaker 194C)
Indicates this proper arrangement. The image 172C is responsive to the user confirming using
the confirm button 178A of the image 170C that the mobile device 18A should participate in
supporting the SL virtual speakers of the collaborative surround sound system 10, It can be
generated and displayed.
[0137]
[0131]
Using the images 170 and / or 172, the user of the collaborative surround sound system can
move the SL speakers of the collaborative surround sound system to the SL speaker area. The
head-end device 14 periodically updates these images as described above to reflect the
movement of the SL speakers in the room configuration and facilitate user repositioning of the SL
speakers be able to. That is, the head end device 14 can cause the speaker to emit the above
mentioned sound continuously, detect this sound, and update the position of this speaker with
respect to the other speakers in the image, here This updated image is then displayed. In this
way, the present technique potentially achieves a more optimal surround sound speaker
configuration that reproduces a more accurate sound stage for a more immersive surround
10-05-2019
47
sound experience. Encourage the adaptation structure of
[0138]
[0132]
8A-8C are flowcharts illustrating exemplary operation of the headend device 14 and the mobile
device 18 in performing various aspects of the collaborative surround sound system techniques
described in this disclosure. Although described below with respect to a particular one of the
mobile devices 18, namely the mobile device 18A in the example of FIG. 5, the present
techniques are mobile devices in a manner similar to that described herein with respect to the
mobile device 18A. It may be performed by 18B-18N.
[0139]
[0133]
Initially, control unit 40 of mobile device 18A may execute cooperative sound system application
42 (210). The collaborative sound system application 42 may first attempt to find the presence
of the headend device 14 on the wireless network (212). If the collaborative sound system
application 42 can not find the headend device 14 on the network ("No" 214), the mobile device
18A can continue to try to find the headend device 14 on the network, while Also, in some cases,
troubleshooting information is presented 212 to assist the user in locating the headend device
14. However, if the collaborative sound system application 42 finds the headend device 14 ("Yes"
214), the collaborative sound system application 42 establishes a session 22A and registers with
the headend device 14 via the session 22A. (216), effectively enabling the headend device 14 to
identify the mobile device 18A as a device that includes the speaker 20A and can participate in
the collaborative surround sound system 10.
[0140]
[0134]
After registering with the headend device 14, the collaborative sound system application 42 can
invoke the data acquisition engine 46, which collects mobile device data 60 in the manner
described above (218). . Data collection engine 46 may then transmit mobile device data 60 to
headend device 14 (220). Data retrieval engine 32 of headend device 14 receives mobile device
data 60 (221) and determines whether the mobile device data 60 includes location data that
10-05-2019
48
defines the location of mobile device 18A relative to headend device 14. (222). If the position
data is not sufficient to allow the headend device 14 to accurately position the mobile device 18A
(such as GPS data with only 30 feet accuracy), or if the position data is mobile device data 60 If
not present ( No 222), the data retrieval engine 32 can call the location module 38, and the
location module 38 calls the location module 48 of the data collection engine 46 called by the
collaborative sound system application 42. And transmit the tone 61 to the location module 48
of the mobile device 18A (224). Location module 48 of mobile device 18A then passes this tone
61 to audio playback module 44, which interfaces with speaker 20A to play tone 61 (226).
[0141]
[0135]
Meanwhile, after transmitting tone 61, location module 38 of headend device 14 can interface
with the microphone to detect playback of tone 61 by speaker 20A (228). Position module 38 of
head end device 14 may then determine 230 the position of mobile device 18A based on the
detected reproduction of tone 61. After determining the location of mobile device 18A using tone
61, data retrieval module 32 of headend device 18 may update mobile device data 60 to include
the determined location, thereby: Generate updated mobile device data 64 (231).
[0142]
[0136]
Head end device 14 may then determine whether to relocate one or more of mobile devices 18 in
the manner described above (FIG. 8B, 232). If the headend device 14 decides to relocate the
mobile device 18A as an example ("Yes" 232), the headend device 14 invokes the image
generation module 160 to execute the process for the display device 164 of the mobile device
18A. One image 170A may be generated 234 and a second image 172A to display device 166 of
source audio device 12 coupled to headend system 14 may be generated 236. The image
generation module 160 may then interface with the display device 164 of the mobile device 18A
to display the first image 170A (238) and simultaneously also to the display of the audio source
device 12 coupled to the head end system 14. The second image 172A is interfaced with the
device 166 (240). The location module 38 of the headend device 14 may determine 242 the
updated current location of the mobile device 18A, where the location module 38 is to be
supported by the mobile device 18A. Based on the intended position of the virtual speaker (such
as the SL virtual speaker shown in the examples of FIGS. 7A-7C) and the updated current
position, determine whether the mobile device 18A is properly positioned. It can do (244).
10-05-2019
49
[0143]
[0137]
If not properly positioned ("No" 244), the headend device 14 displays each display reflecting the
current position of the mobile device 18A relative to the intended position of the virtual speaker
to be supported by the mobile device 18A. Images (such as images 170B and 172B) for display
via 164 and 166 may continue to be generated in the manner described above (234-244). When
properly positioned ("Yes" 244), the headend device 14 will add a mobile device 18A to support a
corresponding one of the virtual surround sound speakers of the cooperative surround sound
system 10. , Can receive confirmation.
[0144]
[0138]
Referring back to FIG. 8B, after relocating one or more of the mobile devices 18, position data
may be present in the mobile device data 60 (or the head end device 14 may be relative to the
head end device 14). If the data retrieval module 32 determines that the mobile device 18 is
accurate enough to enable positioning, or after generating the updated mobile device data 64 to
include the determined position The data retrieval module 32 may then determine 246 whether
it has finished retrieving mobile device data 60 from each of the mobile devices 18 registered
with the headend device 14. If the data retrieval module 32 of the headend device 14 has not
finished retrieving the mobile device data 60 from each of the mobile devices 18 ("No" 246), the
data retrieval module 32 proceeds in the manner described above ( 221-246), continuing with
retrieving mobile device data 60 and generating updated mobile device data 64. However, if the
data retrieval module 32 determines that it has finished collecting the mobile device data 60 and
generating the updated mobile device data 64 ("Yes" 246), the data retrieval module 32 updates
The mobile device data 64 is passed to the audio rendering engine 36.
[0145]
[0139]
Audio rendering engine 36 may retrieve source audio data 37 in response to receiving this
updated mobile device data 64 (248). Audio rendering engine 36 may then render audio signal
66 from source audio data 37 based on mobile device data 64 in the manner described above
when rendering source audio data 37 (250). In some examples, audio rendering engine 36 may
10-05-2019
50
first determine a speaker area, which represents the area in which the speakers should be
arranged to match the playback of multi-channel source audio data 37. For example, 5.1 channel
source audio data includes a front left channel, a center channel, a front light channel, a surround
left channel, a surround light channel, and a subwoofer channel. The subwoofer channel is not
directional or worth noting considering that low frequencies usually provide a sufficient effect
regardless of the position of the subwoofer relative to the head end device. However, the other
five channels may need to be properly positioned to provide the best sound stage for immersive
audio reproduction. The audio rendering engine 36 may interface with the location module 38 to
derive room boundaries in some instances, such that the location module 38 locates walls,
people, furniture, etc. , Speaker 16 and / or speaker 20 may emit tone or sound. Based on the
room or object position information, the audio rendering engine 36 can determine the speaker
area for each of the front left speaker, the center speaker, the front light speaker, the surround
left speaker, and the surround light speaker.
[0146]
[0140]
Based on these speaker areas, the audio rendering engine 36 can determine the position of the
virtual speakers of the collaborative surround sound system 10. That is, the audio rendering
engine 36 can place a virtual speaker in each of the speaker areas that are often at optimal or
near-optimal positions for room or object position information. Audio rendering engine 36 may
then assign mobile device 18 to each virtual speaker based on mobile device data 18.
[0147]
[0141]
For example, the audio rendering engine 36 may first consider the location of each of the mobile
devices 18 defined in the updated mobile device data 60, and those devices are closest to the
determined location of the mobile device 18 Assign to a virtual speaker having a virtual position.
The audio rendering engine 36 may determine whether to assign more than one of the mobile
devices 18 to a virtual speaker based on how close the currently assigned mobile device 18 is to
the position of the virtual speaker. Moreover, when the audio rendering engine 36 determines
that the refined power data 62 associated with one of the two or more mobile devices 18 is
insufficient to reproduce the entire source audio data 37, the mobile device 18 may It can be
decided to assign two or more of them to the same virtual speaker. Audio rendering engine 36
may also assign these mobile devices 18 based on other aspects of mobile device data 60
including speaker characteristics.
10-05-2019
51
[0148]
[0142]
In any event, the audio rendering engine 36 then instantiates or otherwise defines a
preprocessing function to render the audio signal 66 from the source audio data 37, as described
in more detail above. Can. In this manner, audio rendering engine 36 may render source audio
data 37 based on the virtual speaker location and mobile device data 60. As mentioned above,
when the audio rendering engine 36 processes this audio data, it considers the mobile device
data 60 from each of the mobile devices 18 as a whole or as a whole, and still separate audio
signals 66 or one of them. Can be sent to each of the mobile devices 18. Thus, the audio
rendering engine 36 sends 252 the rendered audio signal 66 to the mobile device 18.
[0149]
[0143]
In response to receiving this rendered audio signal 66, the collaborative sound system
application 42 interfaces with the audio playback module 44, which is then rendered interfaced
with the speaker 20A. The audio signal 66 is played (254). As mentioned above, the collaborative
sound system application 42 can periodically call the data collection engine 46 to determine if
any of the mobile device data 60 has changed or has been updated. (256). If the mobile device
data 60 has not changed ( No 256), the mobile device 18A continues to play the rendered
audio signal 66 (254). However, if the mobile device data 60 has changed or is updated ("Yes"
256), the data collection engine 46 sends this modified mobile device data 60 to the data
retrieval engine 32 of the headend device 14. It can be done (258).
[0150]
[0144]
The data retrieval engine 32 can pass this modified mobile device data to the audio rendering
engine 36, which is mobile via the construction of virtual speakers based on the modified mobile
device data 60 The pre-processing function can be modified to process the channel to which
device 18A is assigned. As described in more detail above, mobile device data 60 that is generally
updated or changed may be due to changes in power consumption or for other tasks, such as
voice calls that cause audio playback to be interrupted. Due to the fact that the mobile device
10-05-2019
52
18A has been used before, it changes. In this manner, audio rendering engine 36 may render
audio signal 66 from source audio data 37 based on updated mobile device data 64 (260).
[0151]
[0145]
In some examples, data retrieval engine 32 may determine that mobile device data 60 has
changed in the sense that location module 38 of data retrieval module 32 may detect a change in
the position of mobile device 18A. In other words, the data retrieval module 32 can periodically
call the location module 38 to determine the current location of the mobile device 18 (or
alternatively, the location module 38 continuously locates the location of the mobile device 18
Can be monitored). The location module 38 may then determine if one or more of the mobile
devices 18 have been moved, thereby causing the audio rendering engine 36 to generate
ongoing changes in the location of the mobile devices 18 (eg, the user It allows to dynamically
modify the preprocessing function to get the mobile device to see the text message and then
adapt it to what would happen if you put the mobile device in a different position. Thus, the
present technique may possibly ensure that the virtual speakers stay at least near the optimal
position during the entire playback, even though the mobile device 18 may be moved or
repositioned during playback. It can be applied in a dynamic scene.
[0152]
[0146]
9A-9C are block diagrams illustrating various configurations of cooperative surround sound
systems 270A-270C formed in accordance with the techniques described in this disclosure. FIG.
9A is a block diagram illustrating a first configuration of a cooperative surround sound system
270A. As shown in the example of FIG. 9A, the cooperative surround sound system 270A
includes a source audio device 272, a headend device 274, a front left speaker 276A, a front
light speaker 276B ("speaker 276"), and a speaker 280A. And the mobile device 278A. Each of
the devices and / or speakers 272-278 are of the devices and / or speakers 12-18, as described
above with respect to the examples of FIGS. 1, 2, 3A-3C, 5, 8A-8C. It may be similar to or
substantially similar to the corresponding one.
[0153]
[0147]
10-05-2019
53
The audio rendering engine 36 of the head end device 274 can thus receive the updated mobile
device data 64 including the refined power data 62 in the manner described above. Audio
rendering engine 36 may effectively perform audio distribution using aspects of dynamic
amplitude panning based on constraint vectors of the techniques described in more detail above.
For this reason, audio rendering engine 36 may be referred to as an audio distribution engine.
The audio rendering engine 36 may perform dynamic amplitude panning based on the constraint
vector based on the updated mobile device data 64, including the refined power data 62.
[0154]
[0148]
In the example of FIG. 9A, it is assumed that only a single mobile device 278A participates in the
support of one or more virtual speakers of the collaborative surround sound system 270A. In this
example, there are only two speakers 276 and speaker 280A of the mobile device 278A, which
has been added to the cooperative surround sound system 270A, which is usually not sufficient
to render a 5.1 surround sound format It may be sufficient for other surround sound formats,
such as the Dolby Surround sound format. In this example, it is assumed that the refined power
data 62 indicates that the remaining power of the mobile device 278A is only 30%.
[0155]
[0149]
In assisting the virtual speakers of the cooperative surround sound system 270A to render audio
signals to the speakers, the headend device 274 first performs this refinement in relation to the
duration of the source audio data 37 to be reproduced by the mobile device 278A. Integrated
power data 62 may be considered. To illustrate, when the head end device 274 plays the
allocated one or more channels of the source audio data 37 at maximum volume, the 30% power
level specified by the refined power data 62 is approximately It may be determined that the 30
minutes of source audio data 37 can be played by the mobile device 278A, where this 30
minutes may be called the expected power period. Head end device 274 may then determine that
source audio data 37 has a source audio duration of 50 minutes. Comparing this source audio
period with the expected power period, the audio rendering engine 36 of the headend device 274
renders the source audio data 37 using dynamic amplitude panning based on the constraint
vector to obtain the mobile device 278A. An audio signal can be generated for playback by,
which extends the expected power period so that the source audio period can be exceeded. As an
example, the audio rendering engine 36 may determine that the expected power period extends
10-05-2019
54
to about 60 minutes by lowering the volume by 6 dB. As a result, the audio rendering engine 36
may define a preprocessing function to render the audio signal 66 to the mobile device 278A,
adjusted to be 6 dB lower with respect to volume.
[0156]
[0150]
The audio rendering engine 36 may periodically or continuously monitor the expected power
duration of the mobile device 278A, enabling the mobile device 278A to be able to play back the
entire source audio data 37. Update or redefine the preprocessing function. In some examples,
the user of mobile device 278A can define preferences that define upper limits or other measures
for power levels. That is, the user interfaces with the mobile device 278A and, by way of example,
requires that the mobile device 278A remain at least a certain amount of power, for example
50%, after the playback of the source audio data 37 is complete can do. The user does not have
to charge the mobile device 278A after the playback of the source audio data 37, the mobile
device 278A has other purposes (e.g. emergency purpose, call, email, text messaging, location
guidance using GPS) Etc.), it may be desirable to set such power preferences.
[0157]
[0151]
FIG. 9B shows the cooperative surround sound system shown in the example of FIG. 9A, except
that the cooperative surround sound system 270B includes two mobile devices 278A, 278B, each
of which includes speakers (speakers 280A and 280B, respectively). FIG. 16 is a block diagram
illustrating another configuration of a cooperative surround sound system 270B substantially
similar to 270A. In the example of FIG. 9B, the audio rendering engine 36 of the headend device
274 indicates that only 20% battery power remains on the mobile device 278A, but 100% battery
power remains on the mobile device 278B. It is assumed that the refined power data 62 has been
received. As described above, audio rendering engine 36 may compare the expected power
period of mobile device 278 A with the source audio period determined for source audio data 37.
[0158]
[0152]
10-05-2019
55
If the expected power period is shorter than the source audio period, then the audio rendering
engine 36 then sends an audio signal from the source audio data 37 in a manner that allows the
mobile device 278A to play the entire rendered audio signal 66. 66 can be rendered. In the
example of FIG. 9B, the audio rendering engine 36 cross-mixes one or more aspects of the
surround sound left channel of the source audio data 37 with the rendered front left channel of
the source audio data 37. Left channel can be rendered. In some instances, the audio rendering
engine 36 may define a pre-processing function to cross-mix some portions of lower frequencies
of the surround sound left channel with the front left channel, which may be the mobile device
278A. Can effectively enable tweeter to high frequency components. In some instances, the audio
rendering engine 36 cross mixes this surround sound left channel with the front left channel,
reduces the volume, and supports the surround sound left channel in the manner described
above with respect to the example of FIG. 9A Power consumption by the mobile device 278A
may be further reduced while playing back the audio signal 66. In this regard, the audio
rendering engine 36 may process the same channel in an effort to reduce power consumption by
the mobile device 278A while reproducing the audio signal 66 corresponding to one or more
channels of the source audio data 37. One or more different preprocessing functions can be
applied.
[0159]
[0153]
FIG. 9C shows the cooperative surround sound system shown in the example of FIG. 9A except
that the cooperative surround sound system 270C includes three mobile devices 278A to 278C
each including a speaker (speaker 280A to 280C, respectively). FIG. 16 is a block diagram
illustrating another configuration of a cooperative surround sound system 270C substantially
similar to the cooperative surround sound system 270B shown in the example of 270A and FIG.
9B. In the example of FIG. 9C, the audio rendering engine 36 of the headend device 274 has 90%
battery power remaining in the mobile device 278A while 20% battery power remains in the
mobile device 278B. Is assumed to have received refined power data 62, which indicates that
100% battery power remains. As described above, audio rendering engine 36 may compare the
expected power period of mobile device 278 B to the source audio period determined for source
audio data 37.
[0160]
[0154]
If the expected power period is shorter than the source audio period, then the audio rendering
10-05-2019
56
engine 36 then sends an audio signal from the source audio data 37 in a manner that allows the
mobile device 278B to play the entire rendered audio signal 66. 66 can be rendered. In the
example of FIG. 9C, audio rendering engine 36 includes one or more aspects of the surround
sound center channel of source audio data 37 as surround sound left channels (associated with
mobile device 278A) and (associated with mobile device 278C). The audio signal 66
corresponding to the surround sound center channel can be rendered to cross mix with the
surround sound light channel of the source audio data 37. In some surround sound formats, such
as 5.1 surround sound format, this surround sound center channel may not exist, in which case
the headend device 274 is a surround sound left virtual speaker and a surround sound light
virtual speaker Mobile device 278B may be registered as an aid to one or both. In this case, the
audio rendering engine 36 of the head end device 274 sends source audio data 37 to the mobile
device 278 B in the manner described above with respect to the aspect of amplitude panning
based on the constraint vector of the above described techniques. While reducing the volume of
the rendered audio signal 66, the volume of the rendered audio signal 66 transmitted to one or
both of the mobile devices 278A and 278C can be increased.
[0161]
[0155]
In some instances, the audio rendering engine 36 may use some of the lower frequencies of the
audio signal 66 associated with the surround sound center channel as one of the audio signals
66 corresponding to the surround sound left channel and the surround sound light channel.
Alternatively, pre-processing functions can be defined that cross-mix with multiples, which can
effectively enable the mobile device 278B to act as a tweeter for high frequency components. In
some instances, the audio rendering engine 36 performs this cross-mix while also reducing the
volume while performing the crossmix in the manner described above with respect to the
example of FIGS. 9A, 9B, and the audio signal 66 corresponding to the surround sound center
channel. The power consumption by the mobile device 278B can be further reduced while
playing the. Again, at this point, the audio rendering engine 36 attempts to reduce the power
consumption by the mobile device 278B while playing back the allocated channel or channels of
the source audio data 37, to render the same channel 1 One or more different preprocessing
functions can be applied.
[0162]
[0156]
FIG. 10 is an exemplary operation of a head end device, such as the head end device 274 shown
10-05-2019
57
in the examples of FIGS. 9A-9C, in implementing various power adaptation aspects of the
techniques described in this disclosure. Is a flowchart showing As described in more detail above,
the data retrieval engine 32 of the headend device 274 receives 290 mobile device data 60 from
the mobile device 278 including power consumption data. Data retrieval module 32 invokes
power processing module 34, which processes the power consumption data to generate refined
power data 62 (292). Power processing module 34 returns this refined power data 62 to data
retrieval module 32, which updates mobile device data 60 to include this refined power data 62,
thereby , Generate updated mobile device data 64.
[0163]
[0157]
Audio rendering engine 36 may receive this updated mobile device data 64, including refined
power data 62. The audio rendering engine 36 may then determine the expected power duration
of the mobile device 278 when playing back the audio signal 66 rendered from the source audio
data 37 based on the refined power data 62 Yes (293). Audio rendering engine 36 may also
determine the source audio period of source audio data 37 (294). Audio rendering engine 36
may then determine if the expected power period exceeds the source audio period for any one of
mobile devices 278 (296). If all of the expected power periods exceed the source audio period
("Yes" 298), the headend device 274 renders the audio signal 66 from the source audio data 37
to match other aspects of the mobile device 278. The rendered audio signal 66 may then be sent
302 to the mobile device 278 for playback.
[0164]
[0158]
However, if at least one of the expected power periods does not exceed the source audio period
("No" 298), audio rendering engine 36 may reduce power demand on the corresponding one or
more mobile devices 278, Audio signal 66 may be rendered from source audio data 37 in the
manner described above (300). Head end device 274 may then transmit the rendered audio
signal 66 to mobile device 18 (302).
[0165]
[0159]
10-05-2019
58
To illustrate these aspects of the technique in more detail, an example of watching a movie, and
some small use cases on how such a system may utilize the knowledge of each device's power
usage think of. As mentioned previously, mobile devices may take various forms such as phones,
tablets, fixed appliances, computers and the like. The central device may also be a smart TV, a
receiver or another mobile device with powerful computing capabilities.
[0166]
[0160]
Aspects of power optimization of the above described techniques are described with respect to
distribution of audio signals. Nevertheless, these techniques can be extended to using the screen
of the mobile device and the drive of the camera flash as an extension of media playback. In this
example, the headend device can learn from the media source and analyze for the possibility of
lighting enhancement. For example, in movies with night thunderstorm scenes, some thunder
may be accompanied by flashes to the surroundings, which may improve the visual experience
and become more immersive. In movies with scenes with candles around the viewer in the
church, the extended candle light source can be rendered on the screen of the mobile device
around the viewer. In this visual field, power analysis and management for collaborative systems
may be similar to the audio context described above.
[0167]
[0161]
FIGS. 11-13 illustrate spherical harmonics based functions of various orders or suborders. These
basis functions may be associated with coefficients, which are arranged in two or three
dimensional sound fields in a manner similar to how discrete cosine transform (DCT) coefficients
may be used to represent a signal. It can be used to represent. The techniques described in this
disclosure may be performed with respect to spherical harmonic coefficients or any other type of
hierarchical element that may be utilized to represent a sound field. The following describes the
development of spherical harmonic coefficients that are used to represent the sound field and
form higher order ambisonics audio data.
[0168]
[0162]
10-05-2019
59
The development of sound fields has made available many output formats for entertainment
today. Examples of such surround sound formats are the general 5.1 format (which is front left
(FL), front light (FR), center or front center, back left or surround left, back light Or surround
light and low frequency effects (LFE), including 6 channels), the evolving 7.1 format, and the
upcoming 22.2 format (for example, for use with ultra high definition television standards)
Including. Another example of a spatial audio format is spherical harmonic coefficients (also
known as higher order ambisonics).
[0169]
[0163]
The input to the future standardized audio encoder (device to convert PCM audio representation
to bit stream-storing the number of bits needed per time sample) is optionally (i) pre-defined
Conventional channel based audio, intended to be played through a loudspeaker in position, (ii)
discrete for a single audio object with associated metadata including position coordinates (among
other things information) One of three possible formats: object-based audio with pulse code
modulation (PCM) data, and (iii) scene-based audio with representing a sound field using
spherical harmonic coefficients (SHC) And the above-mentioned spherical harmonic coefficient is
spherical harmonic based Representing the "weight" of linear addition of the function. In this
context, SHC is also known as higher order Ambisonics signal.
[0170]
[0164]
There are various "surround" formats in the market. They are, for example, from the 5.1 home
theater system (which has succeeded in moving into a living room beyond stereo), from NHK
(Nippon Hoso Kyokai) or Japan Broadcasting Association (Japan Broadcasting) It extends to the
22.2 system developed by Corporation)). Content creators (e.g., Hollywood Studios) want to
create movie soundtracks at one time, and do not want to make an effort to remix the
soundtracks for each speaker configuration. More recently, for standardization committees to
provide coding into standardized bitstreams and subsequent decoding that can be adapted to the
geometry of the speakers and acoustic conditions in the position of the renderer and not depend
on them. I'm thinking of a way.
[0171]
[0165]
10-05-2019
60
A set of hierarchical elements can be used to represent the sound field to provide such flexibility
for content creators. A set of hierarchical elements may refer to a set of elements in which the
elements are ordered such that the basic set of lower order elements provides a complete
representation of the modeled sound field. Since this set is extended to include higher order
elements, the representation is more detailed.
[0172]
[0166]
An example of a hierarchical set of elements is a set of spherical harmonic coefficients (SHC). The
following equation shows a description or expression of a sound field using SHC.
[0173]
This equation shows that the pressure p 1 of the sound field at any point {r r, θ r, φ r} (which is
represented in spherical coordinates for the microphone capturing the sound field in this
example) is SHC
[0174]
Indicates that it can be uniquely represented by
ここで、
[0175]
C is the velocity of sound (˜343 m / s), {r r, θ r, φ r} is the reference point (or observation
point),
[0176]
Is the n-order sphere Bessel function,
10-05-2019
61
[0177]
Is a spherical harmonic based function of order n and suborder m.
The terms in the square brackets may be approximated in the frequency domain of the signal (ie,
S (ie, S (the S), which may be approximated by various time-frequency transforms such as
discrete Fourier transform (DFT), discrete cosine transform (DCT), or wavelet transform). It can
be appreciated that ω, r r, θ r, φ r)).
Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets
of coefficients of multi-resolution based functions.
[0178]
[0167]
FIG. 11 is a diagram showing a zeroth-order spherical harmonic based function 410, a first-order
spherical harmonic based function 412A-412C, and a second-order spherical harmonic based
function 414A-414E. The order is specified by the rows of the table, shown as rows 416A-416C,
where row 416A points to the zeros, row 416B points to the primary, and row 416C points to the
secondary. The suborders are identified by the columns of the table shown as columns 418A418E, with column 418A pointing to the 0th suborder, column 418B pointing to the 1st
suborder, and column 418C the -1 suborder Column 418D points to the second order suborder,
and column 418E points to the second order suborder. The SHC corresponding to the zerothorder spherical harmonics-based function 410 can be considered to define the energy of the
sound field, but the remaining higher-order spherical harmonics-based functions (eg, spherical
harmonics-based functions 412A ˜ The SHC corresponding to 412C and 414A-414E) may define
the direction of its energy.
[0179]
[0168]
FIG. 2 is a diagram showing a spherical harmonics based function from 0th order (n = 0) to 4th
order (n = 4). As can be seen, for each order, there is an extension of a suborder m, which is
shown but not explicitly mentioned in the example of FIG. 2 for the sake of simplicity.
10-05-2019
62
[0180]
[0169]
FIG. 3 is another diagram showing a spherical harmonics based function from 0th order (n = 0) to
4th order (n = 4). In FIG. 3, the spherical harmonic based function is shown in a three
dimensional coordinate space with both the indicated order and the suborder.
[0181]
[0170]いずれにしても、SHC
[0182]
Can be physically obtained (eg, recorded) by various microphone array configurations, or
alternatively can be derived from a channel-based or object-based description of the sound field
is there.
SHC represents scene-based audio. For example, the fourth-order SHC representation involves (1
+ 4) <2> = 25 coefficients per time sample.
[0183]
[0171]
To illustrate how these SHCs can be derived from object-based descriptions, consider the
following equations. Coefficients for the sound field corresponding to each audio object
[0184]
は、
[0185]
Can be expressed as, where i is
10-05-2019
63
[0186]
であり、
[0187]
Is the n-th (second type) sphere Hankel function, and {r s, θ s, φ s} is the position of the object.
Knowing the source energy g (ω) as a function of frequency (eg, using time-frequency analysis
techniques such as performing a fast Fourier transform on the PCM stream) SHC each PCM
object and its position
[0188]
Can be converted to
Furthermore, for each object
[0189]
The coefficients may be shown to be additive (as the above equation is linear and orthogonal
decomposition).
In this way, many PCM objects
[0190]
It may be represented by coefficients (eg, as a sum of coefficient vectors for individual objects).
Basically, these coefficients contain information about the sound field (pressure as a function of
10-05-2019
64
3D coordinates) and the above equation gives the whole sound field in the vicinity of the
observation points {r r, θ r, φ r} Represents a transformation from an individual object to a
representation.
[0191]
[0172]The SHC can also be derived from the microphone array recording as follows.
[0192]
ただし、
[0193]
は
[0194]
Is the time domain equivalent of (SHC), * represents the convolution operation, <,> represents the
inner product, b n (ri, t) represents the filter function in the time domain depending on r i, m i (t)
is the signal of the ith microphone, and the ith microphone transducer is located at radius r i,
elevation angle θ i and azimuth angle φ i.
Thus, if there are 32 transducers in the microphone array, and each microphone is placed on the
sphere such that r i = a is constant (like a microphone on an Eigenmike EM32 device with
mhAcoustics), 25 SHCs may be derived using matrix operations as follows.
[0195]
[1] The matrix of the above equation may be more commonly referred to as Es (θ, φ), and the
subscript s is for the set s of transducer geometries with that matrix It can indicate that it is.
The convolution of the above equation (indicated by *) is row-by-row based, so for example,
output
10-05-2019
65
[0196]
Is the vector multiplication of the first row of the Es (θ, φ) matrix with the columns of
microphone signals (which is a function of time and is the cause of the fact that the result of the
vector multiplication is a time series) It is the result of convolution of the time series obtained
and b 0 (a, t).
[0197]
[0173]
The techniques described in this disclosure may be implemented for these spherical harmonic
coefficients.
To illustrate, the audio rendering engine 36 of the head end device 14 shown in the example of
FIG. 2 can render audio signals 66 from source audio data 37 that may define these SHCs.
The audio rendering engine 36 may be more completely and / or exactly during playback if the
SHC may describe the sound field more completely and / or more accurately than object-based
audio data or channel-based audio data. Various transformations can be performed to reproduce
the sound field so as to render various audio signals 66 that can reproduce the sound field, which
causes the position of the speaker 16 and / or the speaker 20 there is a possibility. Moreover,
assuming that the sound field is often more accurately and more completely represented using
SHC, the audio rendering engine 36 will adapt the audio signal 66 to any position of the majority
of the speakers 16 and 20. Can be generated. SHC is relative to the position of the loudspeaker,
which exists in most any standard surround sound format or multi-channel audio format
(including 5.1, 7.1 and 22.2 surround sound formats mentioned above) Constraints can be
effectively eliminated.
[0198]
[0174]
Depending on the example, some operations or events of any of the methods described herein
may be performed in a different order and may be added to, integrated with, or excluded from
each other (e.g. all It should be understood that the operations or events described are not
necessary for the implementation of the method). Moreover, in some instances, operations or
10-05-2019
66
events may be performed simultaneously rather than sequentially, eg, through multi-threaded
processing, interrupt processing, or multiple processors. Further, although some aspects of the
disclosure are described as being implemented by a single module or unit for clarity, the
techniques of the disclosure are implemented by a combination of units or modules associated
with a video coder. Understand that it can be done.
[0199]
[0175]
In one or more examples, the functions described may be implemented in hardware, software,
firmware, or any combination thereof. When implemented in software, the functions may be
stored as one or more instructions or code on a computer readable medium, or may be
transmitted via the computer readable medium and executed by a hardware based processing
unit. Computer-readable media includes, for example, computer-readable storage media
corresponding to tangible media, such as data storage media or communication media, including
any media that facilitates transfer of a computer program from one place to another according to
a communication protocol. May be included.
[0200]
[0176]
In this way, the computer readable medium may generally correspond to (1) a tangible computer
readable storage medium that is non-transitory, or (2) a communication medium such as a signal
or carrier wave. A data storage medium is any use that can be accessed by one or more
computers or one or more processors to retrieve instructions, code and / or data structures for
implementation of the techniques described in this disclosure. It may be a possible medium. A
computer program product may include computer readable media.
[0201]
[0177]
By way of example and not limitation, such computer readable storage media may be RAM, ROM,
EEPROM®, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic
storage device, flash memory or There may be any other medium used to store desired program
code in the form of instructions or data structures and which can be accessed by a computer.
10-05-2019
67
Also, any connection is properly termed a computer-readable medium. For example, instructions
may be sent from a website, server, or other remote source using coaxial cable, fiber optic cable,
twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, wireless, and
microwave. When included, wireless technology such as coaxial cable, fiber optic cable, twisted
pair, DSL, or infrared, wireless, and microwave are included in the definition of medium.
[0202]
[0178]
However, it should be understood that computer readable storage media and data storage media
do not include connections, carriers, signals, or other temporary media, but instead are directed
to non-transitory tangible storage media. As used herein, disks and discs are compact discs (CDs),
laser discs (registered trademark) (discs), optical discs (discs), digital versatile discs (discs) DVDs,
including floppy disks and Blu-ray disks, which typically reproduce data magnetically and disks
laser data. Reproduce optically with Combinations of the above should also be included within
the scope of computer readable media.
[0203]
[0179]
The instructions may be one or more processors, such as one or more digital signal processors
(DSPs), a general purpose microprocessor, an application specific integrated circuit (ASIC), a field
programmable logic array (FPGA), or other equivalent integration. It may be implemented by
circuitry or discrete logic circuitry. Thus, the term "processor" as used herein may refer to any of
the foregoing structure or any other structure suitable for implementation of the techniques
described herein. Further, in some aspects, the functionality described herein may be provided
within dedicated hardware and / or software modules configured for encoding and decoding, or
may be incorporated into a composite codec. Also, the techniques may be fully implemented in
one or more circuits or logic elements.
[0204]
[0180]
The techniques of this disclosure may be implemented in a wide variety of devices or
apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (eg, a chip set).
10-05-2019
68
Although this disclosure describes various components, modules, or units to highlight functional
aspects of devices configured to perform the disclosed techniques, those components, modules,
or units may be Not necessarily need to be implemented by different hardware units. Rather, as
described above, the various units may be combined or interworking hardware in the codec
hardware unit, including one or more processors as described above, along with suitable
software and / or firmware. It can be given by a set of units.
[0205]
[0181]
Various embodiments of the present technique have been described. These and other
embodiments are within the scope of the following claims.
10-05-2019
69